[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:12 dist-test-slave-1xrh krb5kdc[23785](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:12 dist-test-slave-1xrh krb5kdc[23785](info): set up 2 sockets
Aug 10 23:03:12 dist-test-slave-1xrh krb5kdc[23785](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 23:03:13.818845 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.139s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:13 dist-test-slave-1xrh krb5kdc[23785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:13.944131 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33337
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33337
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:14.025575 23799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:14.025795 23799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:14.025842 23799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:14.026981 23799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:14.027045 23799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:14.027070 23799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:14.027099 23799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:14.027140 23799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:14.028646 23799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:14.028925 23799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:14.029160 23799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:14.034389 23804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:14.034399 23807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:14.034428 23799 server_base.cc:1034] running on GCE node
W20260810 23:03:14.034399 23805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:14.034808 23799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:14.035066 23799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:14.036208 23799 hybrid_clock.cc:648] HybridClock initialized: now 1786402994036193 us; error 86 us; skew 500 ppm
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:14.038697 23799 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:14.039179 23799 webserver.cc:460] Webserver started at http://127.23.56.126:34023/ using document root <none> and password file <none>
I20260810 23:03:14.039429 23799 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:14.039494 23799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:14.039640 23799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:14.040735 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "ff0ed08b0ba84fcc995379dc40a94f26"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "5b447f01ddb9a2554d3f7b656439ecdc"
server_key_iv: "89dc9d5ddc57c6082d3fc95317fd4c42"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.041100 23799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "ff0ed08b0ba84fcc995379dc40a94f26"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "5b447f01ddb9a2554d3f7b656439ecdc"
server_key_iv: "89dc9d5ddc57c6082d3fc95317fd4c42"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.042443 23799 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:14.043196 23815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.043370 23799 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:14.043467 23799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "ff0ed08b0ba84fcc995379dc40a94f26"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "5b447f01ddb9a2554d3f7b656439ecdc"
server_key_iv: "89dc9d5ddc57c6082d3fc95317fd4c42"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.043550 23799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:14.056962 23799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:14.059013 23799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:14.059202 23799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:14.062878 23799 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33337
I20260810 23:03:14.062940 23874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33337 every 8 connection(s)
I20260810 23:03:14.063318 23799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:14.064222 23875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:14.067049 23875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26: Bootstrap starting.
I20260810 23:03:14.068286 23875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:14.069114 23875 log.cc:824] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:14.069130 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 23799
I20260810 23:03:14.069240 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:14.069983 23777 external_mini_cluster.cc:1428] Setting key 716e552bf793887f6715514f4e13c6f6
I20260810 23:03:14.070240 23875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26: No bootstrap required, opened a new log
I20260810 23:03:14.075538 23875 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0ed08b0ba84fcc995379dc40a94f26" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33337 } }
I20260810 23:03:14.075766 23875 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:14.075832 23875 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff0ed08b0ba84fcc995379dc40a94f26, State: Initialized, Role: FOLLOWER
I20260810 23:03:14.075982 23875 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0ed08b0ba84fcc995379dc40a94f26" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33337 } }
I20260810 23:03:14.076085 23875 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:14.076135 23875 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:14.076211 23875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:14.077383 23875 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0ed08b0ba84fcc995379dc40a94f26" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33337 } }
I20260810 23:03:14.077483 23875 leader_election.cc:302] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff0ed08b0ba84fcc995379dc40a94f26; no voters:
I20260810 23:03:14.077702 23875 leader_election.cc:288] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:14.077776 23881 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:14.078003 23881 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [term 1 LEADER]: Becoming Leader. State: Replica: ff0ed08b0ba84fcc995379dc40a94f26, State: Running, Role: LEADER
I20260810 23:03:14.078088 23875 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:14.078100 23881 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0ed08b0ba84fcc995379dc40a94f26" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33337 } }
I20260810 23:03:14.079701 23883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff0ed08b0ba84fcc995379dc40a94f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0ed08b0ba84fcc995379dc40a94f26" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33337 } } }
I20260810 23:03:14.079926 23883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:14.078953 23882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff0ed08b0ba84fcc995379dc40a94f26. Latest consensus state: current_term: 1 leader_uuid: "ff0ed08b0ba84fcc995379dc40a94f26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff0ed08b0ba84fcc995379dc40a94f26" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33337 } } }
I20260810 23:03:14.081125 23882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:14.081812 23888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:14.082587 23888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:14.084784 23888 catalog_manager.cc:1347] Generated new cluster ID: 911b2f8020934160893c5732849fff55
I20260810 23:03:14.084848 23888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402993, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:14.097051 23888 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:14.097780 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:14.071038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.23.56.126:33337)
0810 23:03:14.071351 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:14.071359 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:14.071565 (+ 206us) server_negotiation.cc:407] Connection header received
0810 23:03:14.076125 (+ 4560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:14.076133 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:14.076188 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:14.076320 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:14.078551 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.081375 (+ 2824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.083273 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.084230 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.084349 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:14.091069 (+ 6720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:14.091074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:14.091075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:14.091088 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:14.094587 (+ 3499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.095294 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.095298 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.095299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.095554 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.095944 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.095945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.095945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.096395 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:14.096694 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:14.097234 (+ 540us) server_negotiation.cc:299] Negotiation successful
0810 23:03:14.097422 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":219,"threads_started":1}
I20260810 23:03:14.097821 23888 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:14.101308 23888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff0ed08b0ba84fcc995379dc40a94f26: Generated new TSK 0
I20260810 23:03:14.101461 23888 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:14.134348 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33337
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:14.216720 23903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:14.216943 23903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:14.216984 23903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:14.218194 23903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:14.218288 23903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:14.218333 23903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:14.219822 23903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:14.220082 23903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:14.220310 23903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:14.225741 23911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:14.225741 23909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:14.225980 23908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:14.226572 23903 server_base.cc:1034] running on GCE node
I20260810 23:03:14.226801 23903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:14.227044 23903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:14.228204 23903 hybrid_clock.cc:648] HybridClock initialized: now 1786402994228191 us; error 32 us; skew 500 ppm
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:14.231082 23903 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:14.231642 23903 webserver.cc:460] Webserver started at http://127.23.56.65:44735/ using document root <none> and password file <none>
I20260810 23:03:14.231917 23903 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:14.231976 23903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:14.232111 23903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:14.233822 23903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "bf24e9d4c4724e638ff034f9ef895fe1"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "0214a2976f2f6f6c2a07cc98ef42560b"
server_key_iv: "8bda470a78e5d39f80d3c792243308ef"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.234288 23903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "bf24e9d4c4724e638ff034f9ef895fe1"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "0214a2976f2f6f6c2a07cc98ef42560b"
server_key_iv: "8bda470a78e5d39f80d3c792243308ef"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.236282 23903 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 23:03:14.237188 23918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.237361 23903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:14.237462 23903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "bf24e9d4c4724e638ff034f9ef895fe1"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "0214a2976f2f6f6c2a07cc98ef42560b"
server_key_iv: "8bda470a78e5d39f80d3c792243308ef"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.237546 23903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:14.256551 23903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:14.258581 23903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:14.258739 23903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:14.258985 23903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:14.259367 23903 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:14.259414 23903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.259438 23903 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:14.259479 23903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.272310 23903 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:42673
I20260810 23:03:14.272459 24049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:42673 every 8 connection(s)
I20260810 23:03:14.272785 23903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:14.280472 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 23903
I20260810 23:03:14.280584 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:14.280740 23777 external_mini_cluster.cc:1428] Setting key 283e88bd45054546002de6b2c5687c21
I20260810 23:03:14.284376 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:14.273637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:37601 (local address 127.23.56.126:33337)
0810 23:03:14.273709 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:14.273711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:14.274053 (+ 342us) server_negotiation.cc:407] Connection header received
0810 23:03:14.274746 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:14.274747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:14.274760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:14.274846 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:14.276146 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.276662 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.277796 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.278045 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.278118 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:14.280804 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:14.280806 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:14.280806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:14.280816 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:14.282558 (+ 1742us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.283043 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.283044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.283045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.283112 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.283456 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.283457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.283458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.283618 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:14.283682 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:14.284190 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 23:03:14.284295 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:03:14.284911 24052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:14.273608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33337 (local address 127.23.56.65:37601)
0810 23:03:14.273897 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:14.273938 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:14.274634 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:14.274929 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:14.274932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:14.275744 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:14.276044 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:14.276049 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.276755 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.276756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:14.277701 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:14.277703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.278148 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.278149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:14.278328 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:14.278927 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:14.278937 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:14.280706 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:14.282659 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:14.282666 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:14.282669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:14.282968 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:14.283202 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:14.283202 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:14.283203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:14.283336 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:14.283748 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:14.283754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:14.283954 (+ 200us) client_negotiation.cc:769] Sending connection context
0810 23:03:14.284495 (+ 541us) client_negotiation.cc:240] Negotiation successful
0810 23:03:14.284680 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":181,"threads_started":1}
I20260810 23:03:14.285564 24050 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33337
I20260810 23:03:14.285660 24050 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:14.285853 24050 heartbeater.cc:511] Master 127.23.56.126:33337 requested a full tablet report, sending...
I20260810 23:03:14.286427 23834 ts_manager.cc:194] Registered new tserver with Master: bf24e9d4c4724e638ff034f9ef895fe1 (127.23.56.65:42673)
I20260810 23:03:14.287463 23834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:37601
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:14.309893 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33337
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:14.394299 24055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:14.394547 24055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:14.394582 24055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:14.395725 24055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:14.395797 24055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:14.395846 24055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:14.397274 24055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:14.397523 24055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:14.397732 24055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:14.402611 24064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:14.402629 24061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:14.402871 24055 server_base.cc:1034] running on GCE node
W20260810 23:03:14.402874 24062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:14.403203 24055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:14.403432 24055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:14.404583 24055 hybrid_clock.cc:648] HybridClock initialized: now 1786402994404576 us; error 34 us; skew 500 ppm
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:14.407044 24055 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:14.407498 24055 webserver.cc:460] Webserver started at http://127.23.56.66:34627/ using document root <none> and password file <none>
I20260810 23:03:14.407723 24055 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:14.407788 24055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:14.407922 24055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:14.408955 24055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "468235507a724ac884412423e71deee8"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "00f6c35a35e8f80f65a1e2252758fbab"
server_key_iv: "d380b1307cf11dadfcbd517419dc7098"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.409349 24055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "468235507a724ac884412423e71deee8"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "00f6c35a35e8f80f65a1e2252758fbab"
server_key_iv: "d380b1307cf11dadfcbd517419dc7098"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.410629 24055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:14.411339 24072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.411502 24055 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.411598 24055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "468235507a724ac884412423e71deee8"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "00f6c35a35e8f80f65a1e2252758fbab"
server_key_iv: "d380b1307cf11dadfcbd517419dc7098"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.411664 24055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:14.429595 24055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:14.431829 24055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:14.431998 24055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:14.432279 24055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:14.432611 24055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:14.432672 24055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.432715 24055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:14.432742 24055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.443852 24055 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:39723
I20260810 23:03:14.443981 24204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:39723 every 8 connection(s)
I20260810 23:03:14.444368 24055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:14.445636 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 24055
I20260810 23:03:14.445741 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:14.445921 23777 external_mini_cluster.cc:1428] Setting key 2adce9701fc2d2254f8bc80f0d72d181
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
I20260810 23:03:14.455813 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:14.445185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:50309 (local address 127.23.56.126:33337)
0810 23:03:14.445255 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:14.445257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:14.445625 (+ 368us) server_negotiation.cc:407] Connection header received
0810 23:03:14.446402 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:14.446403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:14.446413 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:14.446449 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:14.447855 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.448345 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.449651 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.449997 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.450073 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:14.452888 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:14.452890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:14.452890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:14.452899 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:14.454322 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.454729 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.454730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.454731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.454789 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.455040 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.455041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.455041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.455229 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:14.455302 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:14.455630 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 23:03:14.455718 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:03:14.456074 24209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:14.445172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33337 (local address 127.23.56.66:50309)
0810 23:03:14.445467 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:14.445521 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:14.446304 (+ 783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:14.446534 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:14.446538 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:14.447447 (+ 909us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:14.447752 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:14.447759 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.448444 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.448445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:14.449553 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:14.449556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.450095 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.450096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:14.450285 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:14.450927 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:14.450936 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:14.452788 (+ 1852us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:14.454420 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:14.454428 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:14.454430 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:14.454654 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:14.454855 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:14.454855 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:14.454856 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:14.454967 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:14.455293 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:14.455300 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:14.455470 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 23:03:14.455596 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 23:03:14.455764 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":181,"threads_started":1}
I20260810 23:03:14.456676 24207 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33337
I20260810 23:03:14.456768 24207 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:14.456955 24207 heartbeater.cc:511] Master 127.23.56.126:33337 requested a full tablet report, sending...
I20260810 23:03:14.457355 23834 ts_manager.cc:194] Registered new tserver with Master: 468235507a724ac884412423e71deee8 (127.23.56.66:39723)
I20260810 23:03:14.457717 23834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:50309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:14.474784 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33337
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:32931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:14.559912 24213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:14.560105 24213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:14.560129 24213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:14.561285 24213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:14.561331 24213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:14.561357 24213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:14.562764 24213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:32931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33337
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:14.563004 24213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:14.563262 24213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:14.568969 24218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:14.569018 24219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:14.568953 24221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:14.569157 24213 server_base.cc:1034] running on GCE node
I20260810 23:03:14.569353 24213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:14.569586 24213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:14.570742 24213 hybrid_clock.cc:648] HybridClock initialized: now 1786402994570725 us; error 29 us; skew 500 ppm
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:14.573273 24213 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:14.573698 24213 webserver.cc:460] Webserver started at http://127.23.56.67:38511/ using document root <none> and password file <none>
I20260810 23:03:14.573884 24213 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:14.573925 24213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:14.574023 24213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:14.575013 24213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "df14d9c583fe45ec9ef9034db8fb5632"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "29658dbbb4b620647309e944645929a0"
server_key_iv: "228637a63e84942ff54fbb2347c31ecd"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.575436 24213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "df14d9c583fe45ec9ef9034db8fb5632"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "29658dbbb4b620647309e944645929a0"
server_key_iv: "228637a63e84942ff54fbb2347c31ecd"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.576817 24213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:14.577520 24229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.577708 24213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.577800 24213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "df14d9c583fe45ec9ef9034db8fb5632"
format_stamp: "Formatted at 2026-08-10 23:03:14 on dist-test-slave-1xrh"
server_key: "29658dbbb4b620647309e944645929a0"
server_key_iv: "228637a63e84942ff54fbb2347c31ecd"
server_key_version: "encryptionkey@0"
I20260810 23:03:14.577855 24213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:14.597776 24213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:14.600117 24213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:14.600272 24213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:14.600670 24213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:14.601043 24213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:14.601080 24213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.601121 24213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:14.601146 24213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:14.607300 24213 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:37153
I20260810 23:03:14.607374 24371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:37153 every 8 connection(s)
I20260810 23:03:14.607698 24213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:14.610481 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 24213
I20260810 23:03:14.610574 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:14.610725 23777 external_mini_cluster.cc:1428] Setting key 034fa7919e9c0a4e5923c36e4e73038a
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:14.616791 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:14.608413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:34677 (local address 127.23.56.126:33337)
0810 23:03:14.608457 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:14.608459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:14.608641 (+ 182us) server_negotiation.cc:407] Connection header received
0810 23:03:14.609165 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:14.609167 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:14.609180 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:14.609259 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:14.610185 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.610758 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.612171 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.612354 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.612409 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:14.614354 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:14.614355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:14.614355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:14.614361 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:14.615743 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.616087 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.616087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.616088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.616135 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.616327 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.616327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.616328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.616419 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:14.616467 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:14.616673 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 23:03:14.616729 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:03:14.617002 24375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:14.608376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33337 (local address 127.23.56.67:34677)
0810 23:03:14.608515 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:14.608558 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:14.609066 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:14.609327 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:14.609331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:14.609893 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:14.610088 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:14.610092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.610852 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.610853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:14.612086 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:14.612089 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.612422 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.612422 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:14.612517 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:14.612920 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:14.612928 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:14.614276 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:14.615813 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:14.615818 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:14.615820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:14.616027 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:14.616189 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:14.616190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:14.616190 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:14.616279 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:14.616467 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:14.616471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:14.616572 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 23:03:14.616649 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:03:14.616757 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260810 23:03:14.617421 24372 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33337
I20260810 23:03:14.617517 24372 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:14.617689 24372 heartbeater.cc:511] Master 127.23.56.126:33337 requested a full tablet report, sending...
I20260810 23:03:14.618033 23834 ts_manager.cc:194] Registered new tserver with Master: df14d9c583fe45ec9ef9034db8fb5632 (127.23.56.67:37153)
I20260810 23:03:14.618410 23834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:34677
I20260810 23:03:14.622629 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:14.646158 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:14.638893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35952 (local address 127.23.56.65:42673)
0810 23:03:14.639098 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:14.639103 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:14.639116 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:03:14.639186 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:14.639188 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:14.639202 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:14.639272 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:14.639832 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.640556 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.641287 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.641602 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.641656 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:14.643804 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:14.643812 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:14.643812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:14.643819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:14.645133 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.645431 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.645435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.645435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.645497 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.645686 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.645687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.645687 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.645815 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:14.645966 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:14.646013 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 23:03:14.646077 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":129,"threads_started":1}
W20260810 23:03:14.646530 23952 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35952
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[23785](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:14.669312 24380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:14.662258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35968 (local address 127.23.56.65:42673)
0810 23:03:14.662331 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:14.662333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:14.662406 (+ 73us) server_negotiation.cc:407] Connection header received
0810 23:03:14.662507 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:14.662508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:14.662520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:14.662584 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:14.663178 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.663770 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.664493 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:14.664719 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:14.664792 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:14.666978 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:14.666980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:14.666981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:14.666989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:14.668215 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.668551 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.668553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.668553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.668607 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:14.668804 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:14.668805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:14.668805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:14.668905 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:14.669036 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:14.669117 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 23:03:14.669201 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:03:14.669866 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 23903
I20260810 23:03:14.676923 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 24055
I20260810 23:03:14.682770 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 24213
I20260810 23:03:14.688144 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 23799
2026-08-10T23:03:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2035 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[24388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[24388](info): set up 2 sockets
Aug 10 23:03:14 dist-test-slave-1xrh krb5kdc[24388](info): commencing operation
krb5kdc: starting...
W20260810 23:03:15.796093 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:15 dist-test-slave-1xrh krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:15.914426 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45191
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:45191
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:15.997447 24402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:15.997673 24402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:15.997712 24402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:15.998873 24402 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:15.998937 24402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:15.998965 24402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:15.998994 24402 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:15.999017 24402 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:16.000547 24402 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:45191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24402
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:16.000823 24402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:16.001042 24402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:03:16.005960 24402 server_base.cc:1034] running on GCE node
W20260810 23:03:16.005918 24408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:16.005957 24411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:16.005908 24407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:16.006395 24402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:16.006698 24402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:16.007853 24402 hybrid_clock.cc:648] HybridClock initialized: now 1786402996007831 us; error 32 us; skew 500 ppm
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:16.010358 24402 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:16.010828 24402 webserver.cc:460] Webserver started at http://127.23.56.126:43933/ using document root <none> and password file <none>
I20260810 23:03:16.011071 24402 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:16.011116 24402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:16.011307 24402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:16.012432 24402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "0ac0a965d69d4b12a98f7eb766299b62"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "02db734d9f6cbc09e78ebd99487d2caa"
server_key_iv: "fa56a3855ad94b1d00d8e524893f479f"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.012790 24402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "0ac0a965d69d4b12a98f7eb766299b62"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "02db734d9f6cbc09e78ebd99487d2caa"
server_key_iv: "fa56a3855ad94b1d00d8e524893f479f"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.014088 24402 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:16.014801 24420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.014976 24402 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:16.015064 24402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "0ac0a965d69d4b12a98f7eb766299b62"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "02db734d9f6cbc09e78ebd99487d2caa"
server_key_iv: "fa56a3855ad94b1d00d8e524893f479f"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.015162 24402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:16.032455 24402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:16.034775 24402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:16.034938 24402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:16.039059 24402 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:45191
I20260810 23:03:16.039098 24482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:45191 every 8 connection(s)
I20260810 23:03:16.039500 24402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:16.040388 24483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:16.042627 24483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62: Bootstrap starting.
I20260810 23:03:16.043229 24483 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:16.043526 24483 log.cc:824] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:16.044211 24483 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62: No bootstrap required, opened a new log
I20260810 23:03:16.045780 24483 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac0a965d69d4b12a98f7eb766299b62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45191 } }
I20260810 23:03:16.045915 24483 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:16.045959 24483 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ac0a965d69d4b12a98f7eb766299b62, State: Initialized, Role: FOLLOWER
I20260810 23:03:16.046123 24483 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac0a965d69d4b12a98f7eb766299b62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45191 } }
I20260810 23:03:16.046186 24483 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:16.046207 24483 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:16.046278 24483 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:16.046789 24483 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac0a965d69d4b12a98f7eb766299b62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45191 } }
I20260810 23:03:16.046897 24483 leader_election.cc:302] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ac0a965d69d4b12a98f7eb766299b62; no voters:
I20260810 23:03:16.047094 24483 leader_election.cc:288] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:16.047170 24487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:16.047351 24487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [term 1 LEADER]: Becoming Leader. State: Replica: 0ac0a965d69d4b12a98f7eb766299b62, State: Running, Role: LEADER
I20260810 23:03:16.047390 24483 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:16.047470 24487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac0a965d69d4b12a98f7eb766299b62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45191 } }
I20260810 23:03:16.047914 24489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ac0a965d69d4b12a98f7eb766299b62. Latest consensus state: current_term: 1 leader_uuid: "0ac0a965d69d4b12a98f7eb766299b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac0a965d69d4b12a98f7eb766299b62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45191 } } }
I20260810 23:03:16.048041 24488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ac0a965d69d4b12a98f7eb766299b62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ac0a965d69d4b12a98f7eb766299b62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45191 } } }
I20260810 23:03:16.048081 24489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:16.048161 24488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:16.048333 24497 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:16.048862 24497 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:16.049084 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 24402
I20260810 23:03:16.049176 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:16.049335 23777 external_mini_cluster.cc:1428] Setting key 28f15967b5469623cda497b362570680
I20260810 23:03:16.050454 24497 catalog_manager.cc:1347] Generated new cluster ID: 2cfad43dc0324df38f0b0da85c5248ea
I20260810 23:03:16.050519 24497 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:16.059773 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.049972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47932 (local address 127.23.56.126:45191)
0810 23:03:16.050177 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.050186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.050218 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:03:16.050342 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.050350 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.050404 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.051326 (+ 922us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:16.051944 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.052866 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.053615 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.054038 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.054146 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.056481 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:16.056484 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:16.056484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:16.056501 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:16.058150 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.058510 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.058513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.058514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.058618 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.058822 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.058823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.058823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.059055 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:16.059247 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.059359 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.059610 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":83,"threads_started":1}
I20260810 23:03:16.077075 24497 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:16.077805 24497 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:16.082552 24497 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ac0a965d69d4b12a98f7eb766299b62: Generated new TSK 0
I20260810 23:03:16.082721 24497 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:16.109696 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45191
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:16.193120 24512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:16.193352 24512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:16.193392 24512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:16.194519 24512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:16.194581 24512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:16.194626 24512 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:16.196182 24512 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24512
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:16.196458 24512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:16.196683 24512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:16.201690 24519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:16.201718 24522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:16.201690 24518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:16.201943 24512 server_base.cc:1034] running on GCE node
I20260810 23:03:16.202134 24512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:16.202421 24512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:16.203555 24512 hybrid_clock.cc:648] HybridClock initialized: now 1786402996203546 us; error 29 us; skew 500 ppm
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:16.205859 24512 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:16.206271 24512 webserver.cc:460] Webserver started at http://127.23.56.65:42057/ using document root <none> and password file <none>
I20260810 23:03:16.206508 24512 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:16.206574 24512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:16.206710 24512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:16.207788 24512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "c6d50a81dc284940bacf9733c4ee9799"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "49d5a7a92f182ed57835c547e1716585"
server_key_iv: "0d8906f4dda67ce4c3160518f7b5375d"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.208176 24512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "c6d50a81dc284940bacf9733c4ee9799"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "49d5a7a92f182ed57835c547e1716585"
server_key_iv: "0d8906f4dda67ce4c3160518f7b5375d"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.209481 24512 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:16.210179 24529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.210345 24512 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:16.210454 24512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "c6d50a81dc284940bacf9733c4ee9799"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "49d5a7a92f182ed57835c547e1716585"
server_key_iv: "0d8906f4dda67ce4c3160518f7b5375d"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.210525 24512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:16.230744 24512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:16.232784 24512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:16.232947 24512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:16.233178 24512 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:16.233487 24512 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:16.233537 24512 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.233578 24512 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:16.233604 24512 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.239337 24512 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:43925
I20260810 23:03:16.239390 24660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:43925 every 8 connection(s)
I20260810 23:03:16.239710 24512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:16.244963 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 24512
I20260810 23:03:16.245042 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:16.245220 23777 external_mini_cluster.cc:1428] Setting key 63ff8d83053204ff521fef6dcb5b4faf
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:16.249279 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.240402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:46551 (local address 127.23.56.126:45191)
0810 23:03:16.240464 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.240466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.240678 (+ 212us) server_negotiation.cc:407] Connection header received
0810 23:03:16.241650 (+ 972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.241651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.241660 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.241752 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:16.242314 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.242808 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.243753 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.243956 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.244010 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.246206 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:16.246208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:16.246208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:16.246216 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:16.247816 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.248322 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.248324 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.248324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.248382 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.248669 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.248670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.248670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.248770 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:16.248821 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.249141 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.249215 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 23:03:16.249663 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.240360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45191 (local address 127.23.56.65:46551)
0810 23:03:16.240554 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:16.240598 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:16.241560 (+ 962us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:16.241810 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:16.241812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:16.242055 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:16.242241 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.242245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.242884 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.242884 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.243681 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.243683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.244029 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.244030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.244123 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.244568 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:16.244576 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:16.246110 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:16.247924 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:16.247932 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:16.247934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:16.248249 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:16.248457 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:16.248458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:16.248458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:16.248605 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:16.248831 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:16.248837 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:16.249003 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 23:03:16.249116 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 23:03:16.249308 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":108,"threads_started":1}
I20260810 23:03:16.250138 24662 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45191
I20260810 23:03:16.250236 24662 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:16.250447 24662 heartbeater.cc:511] Master 127.23.56.126:45191 requested a full tablet report, sending...
I20260810 23:03:16.251009 24440 ts_manager.cc:194] Registered new tserver with Master: c6d50a81dc284940bacf9733c4ee9799 (127.23.56.65:43925)
I20260810 23:03:16.251633 24440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:46551
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:16.272035 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45191
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:16.365779 24667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:16.366008 24667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:16.366051 24667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:16.367249 24667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:16.367312 24667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:16.367357 24667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:16.368826 24667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:16.369102 24667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:16.369319 24667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:16.374161 24673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:16.374171 24676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:16.374171 24672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:16.374373 24667 server_base.cc:1034] running on GCE node
I20260810 23:03:16.374560 24667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:16.374773 24667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:16.375905 24667 hybrid_clock.cc:648] HybridClock initialized: now 1786402996375889 us; error 33 us; skew 500 ppm
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:16.378232 24667 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:16.378661 24667 webserver.cc:460] Webserver started at http://127.23.56.66:37011/ using document root <none> and password file <none>
I20260810 23:03:16.378836 24667 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:16.378875 24667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:16.378961 24667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:16.379961 24667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "66f14c4a1def4b6bb7b9712533c03d90"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "f9b35960b509c67ca917d9b6a004c058"
server_key_iv: "798f968082a99b1d059e5ee2d5dc2eb9"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.380255 24667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "66f14c4a1def4b6bb7b9712533c03d90"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "f9b35960b509c67ca917d9b6a004c058"
server_key_iv: "798f968082a99b1d059e5ee2d5dc2eb9"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.381453 24667 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:16.382085 24684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.382246 24667 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:16.382344 24667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "66f14c4a1def4b6bb7b9712533c03d90"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "f9b35960b509c67ca917d9b6a004c058"
server_key_iv: "798f968082a99b1d059e5ee2d5dc2eb9"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.382412 24667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:16.397405 24667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:16.399371 24667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:16.399556 24667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:16.399807 24667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:16.400137 24667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:16.400193 24667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.400238 24667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:16.400271 24667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.406812 24667 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:41221
I20260810 23:03:16.406860 24813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:41221 every 8 connection(s)
I20260810 23:03:16.407281 24667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:16.416828 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.408045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:37641 (local address 127.23.56.126:45191)
0810 23:03:16.408134 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.408137 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.408331 (+ 194us) server_negotiation.cc:407] Connection header received
0810 23:03:16.409228 (+ 897us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.409229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.409241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.409304 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:16.409903 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.410478 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.411496 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.411688 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.411742 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.413885 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:16.413887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:16.413887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:16.413900 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:16.415320 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.415761 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.415763 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.415763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.415848 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.416115 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.416116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.416117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.416249 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:16.416328 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.416625 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.416717 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:03:16.416831 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 24667
I20260810 23:03:16.416927 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:16.417030 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.408010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45191 (local address 127.23.56.66:37641)
0810 23:03:16.408189 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:16.408223 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:16.409132 (+ 909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:16.409370 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:16.409373 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:16.409623 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:16.409804 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.409808 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.410575 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.410576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.411412 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.411414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.411755 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.411756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.411851 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.412326 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:16.412334 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:16.413737 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:16.415421 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:16.415429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:16.415431 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:16.415664 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:16.415920 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:16.415921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:16.415922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:16.416037 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:16.416322 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:16.416330 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:16.416485 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 23:03:16.416601 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 23:03:16.416771 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":108,"threads_started":1}
I20260810 23:03:16.417140 23777 external_mini_cluster.cc:1428] Setting key d399734a9f23ec56833df39c8a2eea72
I20260810 23:03:16.417681 24814 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45191
I20260810 23:03:16.417790 24814 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:16.418005 24814 heartbeater.cc:511] Master 127.23.56.126:45191 requested a full tablet report, sending...
I20260810 23:03:16.418406 24440 ts_manager.cc:194] Registered new tserver with Master: 66f14c4a1def4b6bb7b9712533c03d90 (127.23.56.66:41221)
I20260810 23:03:16.418810 24440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:37641
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:16.442153 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45191
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41413
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:16.524425 24824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:16.524621 24824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:16.524644 24824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:16.525784 24824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:16.525825 24824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:16.525875 24824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:16.527479 24824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41413
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45191
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:16.527720 24824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:16.527968 24824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:16.532917 24830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:16.533030 24824 server_base.cc:1034] running on GCE node
W20260810 23:03:16.532986 24833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:16.532912 24831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:16.533361 24824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:16.533553 24824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:16.534699 24824 hybrid_clock.cc:648] HybridClock initialized: now 1786402996534679 us; error 33 us; skew 500 ppm
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:16.537369 24824 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:16.537825 24824 webserver.cc:460] Webserver started at http://127.23.56.67:41245/ using document root <none> and password file <none>
I20260810 23:03:16.538046 24824 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:16.538111 24824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:16.538265 24824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:16.539322 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "8c82236f387c482fbcdd33c0e2c729b5"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "a53262aa48396a0becd05a703fecb77f"
server_key_iv: "433fd69ad9de69c861f87b1297457939"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.539675 24824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "8c82236f387c482fbcdd33c0e2c729b5"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "a53262aa48396a0becd05a703fecb77f"
server_key_iv: "433fd69ad9de69c861f87b1297457939"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.540938 24824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:16.541620 24840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.541793 24824 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:16.541909 24824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "8c82236f387c482fbcdd33c0e2c729b5"
format_stamp: "Formatted at 2026-08-10 23:03:16 on dist-test-slave-1xrh"
server_key: "a53262aa48396a0becd05a703fecb77f"
server_key_iv: "433fd69ad9de69c861f87b1297457939"
server_key_version: "encryptionkey@0"
I20260810 23:03:16.541976 24824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:16.552281 24824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:16.553959 24824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:16.554106 24824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:16.554309 24824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:16.554613 24824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:16.554672 24824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.554713 24824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:16.554740 24824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:16.560667 24824 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:41657
I20260810 23:03:16.560711 24972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:41657 every 8 connection(s)
I20260810 23:03:16.561007 24824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:16.567085 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 24824
I20260810 23:03:16.567204 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:16.567358 23777 external_mini_cluster.cc:1428] Setting key 8f18488062134021c6fa705a15c69d55
I20260810 23:03:16.570089 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.561675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:52677 (local address 127.23.56.126:45191)
0810 23:03:16.561729 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.561732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.561900 (+ 168us) server_negotiation.cc:407] Connection header received
0810 23:03:16.562844 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.562846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.562860 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.562911 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:16.563566 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.564112 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.565243 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.565447 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.565523 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.567584 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:16.567586 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:16.567587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:16.567594 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:16.569032 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.569375 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.569376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.569376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.569423 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.569628 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.569629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.569629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.569723 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:16.569771 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.569967 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.570029 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 23:03:16.570362 24975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.561660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45191 (local address 127.23.56.67:52677)
0810 23:03:16.561797 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:16.561830 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:16.562727 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:16.562981 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:16.562985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:16.563288 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:16.563465 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.563470 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.564311 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.564312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.565146 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.565149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.565532 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.565533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.565651 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.566088 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:16.566110 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:16.567487 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:16.569101 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:16.569107 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:16.569108 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:16.569314 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:16.569477 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:16.569478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:16.569478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:16.569577 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:16.569772 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:16.569777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:16.569878 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 23:03:16.569955 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:03:16.570067 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260810 23:03:16.570785 24973 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45191
I20260810 23:03:16.570876 24973 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:16.571044 24973 heartbeater.cc:511] Master 127.23.56.126:45191 requested a full tablet report, sending...
I20260810 23:03:16.571455 24440 ts_manager.cc:194] Registered new tserver with Master: 8c82236f387c482fbcdd33c0e2c729b5 (127.23.56.67:41657)
I20260810 23:03:16.571828 24440 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:52677
I20260810 23:03:16.579294 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:03:16.586491 24507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.580778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.23.56.126:45191)
0810 23:03:16.580815 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.580817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.580914 (+ 97us) server_negotiation.cc:407] Connection header received
0810 23:03:16.580963 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.580963 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.580971 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.581023 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:16.581529 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.581973 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.582699 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.582923 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.583019 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.584139 (+ 1120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:16.584140 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:16.584140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:16.584146 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:16.585482 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.585811 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.585812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.585813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.585875 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.586029 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.586029 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.586030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.586209 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:16.586281 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.586370 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.586428 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 23:03:16.588490 24440 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47936:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:16.590075 24440 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:16.596614 24984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.592264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:41221 (local address 127.0.0.1:60784)
0810 23:03:16.593005 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:16.593012 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:16.593069 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:16.593228 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:16.593228 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:16.593230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:16.593267 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.593267 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.594273 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.594274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.595610 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.595611 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.596309 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.596310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.596390 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.596403 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:03:16.596483 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:03:16.596540 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":733,"mutex_wait_us":2,"thread_start_us":62,"threads_started":1}
I20260810 23:03:16.596630 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.592681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60784 (local address 127.23.56.66:41221)
0810 23:03:16.592819 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.592822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.593072 (+ 250us) server_negotiation.cc:407] Connection header received
0810 23:03:16.593129 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.593132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.593146 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.593218 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:16.593327 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.594170 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.595704 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.596222 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.596309 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.596368 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.596520 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.596558 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260810 23:03:16.596884 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.592263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:43925 (local address 127.0.0.1:46496)
0810 23:03:16.592838 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:16.592868 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:16.592931 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:16.593112 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:16.593115 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:16.593118 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:16.593186 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.593187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.594495 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.594496 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.595608 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.595611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.596621 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.596622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.596702 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.596710 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 23:03:16.596761 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 23:03:16.596804 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":25,"thread_start_us":87,"threads_started":1}
I20260810 23:03:16.596980 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.592510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.23.56.67:41657)
0810 23:03:16.592695 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.592700 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.593511 (+ 811us) server_negotiation.cc:407] Connection header received
0810 23:03:16.593562 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.593564 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.593579 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.593657 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:16.593768 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.594692 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.595800 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.596450 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.596529 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.596580 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.596848 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.596890 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":91,"threads_started":1}
I20260810 23:03:16.597194 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.592476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.23.56.65:43925)
0810 23:03:16.592664 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.592669 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.592916 (+ 247us) server_negotiation.cc:407] Connection header received
0810 23:03:16.592996 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.592998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.593010 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.593107 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:16.593275 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.594395 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.595714 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.596451 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.596525 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.596581 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.597049 (+ 468us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.597105 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260810 23:03:16.597424 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.592266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41657 (local address 127.0.0.1:45366)
0810 23:03:16.593431 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:16.593441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:16.593504 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:16.593664 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:16.593665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:16.593667 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:16.593714 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.593715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.594788 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.594789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.595718 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.595721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.596619 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.596621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.596702 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.596712 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:03:16.596761 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 23:03:16.596802 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1146,"mutex_wait_us":7,"thread_start_us":95,"threads_started":1}
I20260810 23:03:16.597463 24743 tablet_service.cc:1467] Processing CreateTablet for tablet 41371bb66a0e426681b8b011f7c37e6c (DEFAULT_TABLE table=test-table [id=73b1d21138a24e969661ea9de37388b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:16.597859 24743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41371bb66a0e426681b8b011f7c37e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:16.598083 24895 tablet_service.cc:1467] Processing CreateTablet for tablet 41371bb66a0e426681b8b011f7c37e6c (DEFAULT_TABLE table=test-table [id=73b1d21138a24e969661ea9de37388b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:16.598073 24585 tablet_service.cc:1467] Processing CreateTablet for tablet 41371bb66a0e426681b8b011f7c37e6c (DEFAULT_TABLE table=test-table [id=73b1d21138a24e969661ea9de37388b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:16.598392 24895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41371bb66a0e426681b8b011f7c37e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:16.598394 24585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41371bb66a0e426681b8b011f7c37e6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:16.601236 24993 tablet_bootstrap.cc:492] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90: Bootstrap starting.
I20260810 23:03:16.601342 24992 tablet_bootstrap.cc:492] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5: Bootstrap starting.
I20260810 23:03:16.601691 24994 tablet_bootstrap.cc:492] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799: Bootstrap starting.
I20260810 23:03:16.601954 24993 tablet_bootstrap.cc:654] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:16.602032 24992 tablet_bootstrap.cc:654] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:16.602324 24994 tablet_bootstrap.cc:654] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:16.602514 24993 log.cc:824] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:16.602675 24992 log.cc:824] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:16.603076 24994 log.cc:824] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:16.603325 24993 tablet_bootstrap.cc:492] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90: No bootstrap required, opened a new log
I20260810 23:03:16.603403 24993 ts_tablet_manager.cc:1397] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:16.603539 24992 tablet_bootstrap.cc:492] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5: No bootstrap required, opened a new log
I20260810 23:03:16.603605 24992 ts_tablet_manager.cc:1397] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:16.603848 24994 tablet_bootstrap.cc:492] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799: No bootstrap required, opened a new log
I20260810 23:03:16.603904 24994 ts_tablet_manager.cc:1397] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:03:16.604949 24993 raft_consensus.cc:348] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } }
I20260810 23:03:16.605103 24993 raft_consensus.cc:374] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:16.605124 24993 raft_consensus.cc:729] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66f14c4a1def4b6bb7b9712533c03d90, State: Initialized, Role: FOLLOWER
I20260810 23:03:16.605235 24993 consensus_queue.cc:260] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } }
I20260810 23:03:16.605497 24814 heartbeater.cc:503] Master 127.23.56.126:45191 was elected leader, sending a full tablet report...
I20260810 23:03:16.605517 24993 ts_tablet_manager.cc:1428] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:16.605561 24992 raft_consensus.cc:348] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } }
I20260810 23:03:16.605717 24992 raft_consensus.cc:374] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:16.605748 24992 raft_consensus.cc:729] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c82236f387c482fbcdd33c0e2c729b5, State: Initialized, Role: FOLLOWER
I20260810 23:03:16.605865 24992 consensus_queue.cc:260] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } }
I20260810 23:03:16.605822 24994 raft_consensus.cc:348] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } }
I20260810 23:03:16.605968 24994 raft_consensus.cc:374] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:16.606006 24994 raft_consensus.cc:729] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d50a81dc284940bacf9733c4ee9799, State: Initialized, Role: FOLLOWER
I20260810 23:03:16.606137 24992 ts_tablet_manager.cc:1428] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:16.606163 24973 heartbeater.cc:503] Master 127.23.56.126:45191 was elected leader, sending a full tablet report...
I20260810 23:03:16.606137 24994 consensus_queue.cc:260] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } }
I20260810 23:03:16.606392 24994 ts_tablet_manager.cc:1428] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 23:03:16.606693 24662 heartbeater.cc:503] Master 127.23.56.126:45191 was elected leader, sending a full tablet report...
I20260810 23:03:16.628594 25000 raft_consensus.cc:482] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:16.628700 25000 raft_consensus.cc:504] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } }
I20260810 23:03:16.628997 25000 leader_election.cc:288] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66f14c4a1def4b6bb7b9712533c03d90 (127.23.56.66:41221), 8c82236f387c482fbcdd33c0e2c729b5 (127.23.56.67:41657)
I20260810 23:03:16.631829 24991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.629219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:52847 (local address 127.23.56.66:41221)
0810 23:03:16.629276 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.629278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.629289 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:03:16.629320 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.629321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.629334 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.629400 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:16.629607 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.630131 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.631077 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.631476 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.631534 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.631542 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.631711 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.631760 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 23:03:16.631825 24664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.629145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:41221 (local address 127.23.56.65:52847)
0810 23:03:16.629202 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:16.629223 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:16.629266 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:16.629437 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:16.629438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:16.629440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:16.629533 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.629536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.630206 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.630208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.630984 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.630987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.631553 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.631554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.631637 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.631640 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:16.631692 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 23:03:16.631727 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 23:03:16.631887 24990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.629219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:40789 (local address 127.23.56.67:41657)
0810 23:03:16.629270 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.629273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.629436 (+ 163us) server_negotiation.cc:407] Connection header received
0810 23:03:16.629488 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.629490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.629504 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.629562 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:16.629726 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.630200 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.631101 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.631530 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.631603 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.631612 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.631774 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.631812 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:03:16.631996 25001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.629145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41657 (local address 127.23.56.65:40789)
0810 23:03:16.629352 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:16.629362 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:16.629431 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:16.629584 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:16.629585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:16.629587 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:16.629659 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.629660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.630283 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.630284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.631001 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:16.631002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.631617 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.631618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:16.631694 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.631698 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:16.631757 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:03:16.631794 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":15,"thread_start_us":117,"threads_started":1}
I20260810 23:03:16.632189 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41371bb66a0e426681b8b011f7c37e6c" candidate_uuid: "c6d50a81dc284940bacf9733c4ee9799" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f14c4a1def4b6bb7b9712533c03d90" is_pre_election: true
I20260810 23:03:16.632339 24764 raft_consensus.cc:2454] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6d50a81dc284940bacf9733c4ee9799 in term 0.
I20260810 23:03:16.632344 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41371bb66a0e426681b8b011f7c37e6c" candidate_uuid: "c6d50a81dc284940bacf9733c4ee9799" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c82236f387c482fbcdd33c0e2c729b5" is_pre_election: true
I20260810 23:03:16.632506 24918 raft_consensus.cc:2454] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6d50a81dc284940bacf9733c4ee9799 in term 0.
I20260810 23:03:16.632580 24532 leader_election.cc:302] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66f14c4a1def4b6bb7b9712533c03d90, c6d50a81dc284940bacf9733c4ee9799; no voters:
I20260810 23:03:16.632710 25000 raft_consensus.cc:2781] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:16.632774 25000 raft_consensus.cc:482] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:16.632797 25000 raft_consensus.cc:3037] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:16.633440 25000 raft_consensus.cc:504] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } }
I20260810 23:03:16.633579 25000 leader_election.cc:288] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [CANDIDATE]: Term 1 election: Requested vote from peers 66f14c4a1def4b6bb7b9712533c03d90 (127.23.56.66:41221), 8c82236f387c482fbcdd33c0e2c729b5 (127.23.56.67:41657)
I20260810 23:03:16.633728 24918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41371bb66a0e426681b8b011f7c37e6c" candidate_uuid: "c6d50a81dc284940bacf9733c4ee9799" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8c82236f387c482fbcdd33c0e2c729b5"
I20260810 23:03:16.633733 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41371bb66a0e426681b8b011f7c37e6c" candidate_uuid: "c6d50a81dc284940bacf9733c4ee9799" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66f14c4a1def4b6bb7b9712533c03d90"
I20260810 23:03:16.633821 24918 raft_consensus.cc:3037] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:16.633819 24764 raft_consensus.cc:3037] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:16.634505 24918 raft_consensus.cc:2454] T 41371bb66a0e426681b8b011f7c37e6c P 8c82236f387c482fbcdd33c0e2c729b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d50a81dc284940bacf9733c4ee9799 in term 1.
I20260810 23:03:16.634579 24764 raft_consensus.cc:2454] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d50a81dc284940bacf9733c4ee9799 in term 1.
I20260810 23:03:16.634713 24531 leader_election.cc:302] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c82236f387c482fbcdd33c0e2c729b5, c6d50a81dc284940bacf9733c4ee9799; no voters:
I20260810 23:03:16.634799 25000 raft_consensus.cc:2781] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:16.634967 25000 raft_consensus.cc:686] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [term 1 LEADER]: Becoming Leader. State: Replica: c6d50a81dc284940bacf9733c4ee9799, State: Running, Role: LEADER
I20260810 23:03:16.635074 25000 consensus_queue.cc:237] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } }
I20260810 23:03:16.635960 24439 catalog_manager.cc:5495] T 41371bb66a0e426681b8b011f7c37e6c P c6d50a81dc284940bacf9733c4ee9799 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6d50a81dc284940bacf9733c4ee9799 (127.23.56.65). New cstate: current_term: 1 leader_uuid: "c6d50a81dc284940bacf9733c4ee9799" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66f14c4a1def4b6bb7b9712533c03d90" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d50a81dc284940bacf9733c4ee9799" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c82236f387c482fbcdd33c0e2c729b5" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41657 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:16.644703 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.637890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46506 (local address 127.23.56.65:43925)
0810 23:03:16.637956 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.637958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.638021 (+ 63us) server_negotiation.cc:407] Connection header received
0810 23:03:16.638072 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.638072 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.638082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.638124 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:16.638630 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.639046 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.639824 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.640009 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.640062 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.642221 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:16.642229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:16.642229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:16.642235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:16.643645 (+ 1410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.643984 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.643987 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.643987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.644047 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.644206 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.644206 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.644207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.644367 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:16.644495 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.644550 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.644614 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260810 23:03:16.657840 24820 tablet.cc:2369] T 41371bb66a0e426681b8b011f7c37e6c P 66f14c4a1def4b6bb7b9712533c03d90: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:16.668593 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.661951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46514 (local address 127.23.56.65:43925)
0810 23:03:16.662035 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.662037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.662095 (+ 58us) server_negotiation.cc:407] Connection header received
0810 23:03:16.662166 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.662167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.662176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.662218 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:16.662752 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.663274 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.663987 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.664173 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.664224 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.666403 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:16.666404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:16.666404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:16.666412 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:16.667644 (+ 1232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.667966 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.667967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.667967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.668016 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.668167 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.668168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.668168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.668251 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:16.668323 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.668460 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.668523 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[24388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402996, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:16.690951 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:16.684513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.23.56.65:43925)
0810 23:03:16.684577 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:16.684579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:16.684629 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:03:16.684706 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:16.684707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:16.684721 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:16.684784 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:16.685336 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.685852 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.686553 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:16.686733 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:16.686787 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:16.688954 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:16.688955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:16.688956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:16.688962 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:16.690114 (+ 1152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.690396 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.690397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.690397 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.690442 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:16.690578 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:16.690579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:16.690579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:16.690656 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:16.690723 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:16.690822 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 23:03:16.690885 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 23:03:16.691414 24562 tablet_service.cc:2908] Rejecting scan request for tablet 41371bb66a0e426681b8b011f7c37e6c: Uninitialized: clean time has not yet been initialized
I20260810 23:03:16.692305 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 24512
I20260810 23:03:16.699901 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 24667
I20260810 23:03:16.709333 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 24824
I20260810 23:03:16.715960 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 24402
2026-08-10T23:03:16Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2029 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[25018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[25018](info): set up 2 sockets
Aug 10 23:03:16 dist-test-slave-1xrh krb5kdc[25018](info): commencing operation
krb5kdc: starting...
W20260810 23:03:17.823799 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:17 dist-test-slave-1xrh krb5kdc[25018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:17.954447 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45329
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:45329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:18.049901 25034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:18.050128 25034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:18.050169 25034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:18.051311 25034 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:18.051350 25034 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:03:18.051363 25034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:18.051376 25034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:18.051388 25034 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:18.051399 25034 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:18.052748 25034 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:45329
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:18.052940 25034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:18.053114 25034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:18.058065 25039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:18.058064 25042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:18.058161 25040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:18.058221 25034 server_base.cc:1034] running on GCE node
I20260810 23:03:18.058531 25034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:18.058857 25034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:18.060011 25034 hybrid_clock.cc:648] HybridClock initialized: now 1786402998059983 us; error 41 us; skew 500 ppm
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:18.068284 25034 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:18.068781 25034 webserver.cc:460] Webserver started at http://127.23.56.126:45357/ using document root <none> and password file <none>
I20260810 23:03:18.069010 25034 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:18.069075 25034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:18.069223 25034 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:18.070269 25034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "514f9d039f564efd806bca3fabcdbdf3"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "cf46357dc493de2a165ba4f1780d92f2"
server_key_iv: "d6611fc250d794b3bd9acdde6c75ab2b"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.070654 25034 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "514f9d039f564efd806bca3fabcdbdf3"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "cf46357dc493de2a165ba4f1780d92f2"
server_key_iv: "d6611fc250d794b3bd9acdde6c75ab2b"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.072037 25034 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:18.072731 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.072902 25034 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:18.073020 25034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "514f9d039f564efd806bca3fabcdbdf3"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "cf46357dc493de2a165ba4f1780d92f2"
server_key_iv: "d6611fc250d794b3bd9acdde6c75ab2b"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.073091 25034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:18.086621 25034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:18.088593 25034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:18.088747 25034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:18.093698 25034 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:45329
I20260810 23:03:18.093755 25125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:45329 every 8 connection(s)
I20260810 23:03:18.094072 25034 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:18.094668 25126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:18.096982 25126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3: Bootstrap starting.
I20260810 23:03:18.097577 25126 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:18.097855 25126 log.cc:824] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:18.098520 25126 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3: No bootstrap required, opened a new log
I20260810 23:03:18.100734 25126 raft_consensus.cc:348] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514f9d039f564efd806bca3fabcdbdf3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45329 } }
I20260810 23:03:18.100900 25126 raft_consensus.cc:374] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:18.100916 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 25034
I20260810 23:03:18.100946 25126 raft_consensus.cc:729] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 514f9d039f564efd806bca3fabcdbdf3, State: Initialized, Role: FOLLOWER
I20260810 23:03:18.101023 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:18.101107 25126 consensus_queue.cc:260] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514f9d039f564efd806bca3fabcdbdf3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45329 } }
I20260810 23:03:18.101205 25126 raft_consensus.cc:388] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:18.101205 23777 external_mini_cluster.cc:1428] Setting key e56c1f57eeb9f4003c718edb5227b8d8
I20260810 23:03:18.101250 25126 raft_consensus.cc:482] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:18.101318 25126 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:18.102010 25126 raft_consensus.cc:504] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514f9d039f564efd806bca3fabcdbdf3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45329 } }
I20260810 23:03:18.102125 25126 leader_election.cc:302] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 514f9d039f564efd806bca3fabcdbdf3; no voters:
I20260810 23:03:18.102299 25126 leader_election.cc:288] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:18.102341 25131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:18.102526 25131 raft_consensus.cc:686] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [term 1 LEADER]: Becoming Leader. State: Replica: 514f9d039f564efd806bca3fabcdbdf3, State: Running, Role: LEADER
I20260810 23:03:18.102666 25131 consensus_queue.cc:237] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514f9d039f564efd806bca3fabcdbdf3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45329 } }
I20260810 23:03:18.102671 25126 sys_catalog.cc:564] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:18.104020 25132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "514f9d039f564efd806bca3fabcdbdf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514f9d039f564efd806bca3fabcdbdf3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45329 } } }
I20260810 23:03:18.104136 25132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:18.104360 25148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:18.104516 25133 sys_catalog.cc:455] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 514f9d039f564efd806bca3fabcdbdf3. Latest consensus state: current_term: 1 leader_uuid: "514f9d039f564efd806bca3fabcdbdf3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "514f9d039f564efd806bca3fabcdbdf3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45329 } } }
I20260810 23:03:18.104589 25133 sys_catalog.cc:458] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:18.104933 25148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:18.106688 25148 catalog_manager.cc:1347] Generated new cluster ID: d7080e298e3846f99785cb2125b601e2
I20260810 23:03:18.106758 25148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402997, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:18.112666 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.101742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44814 (local address 127.23.56.126:45329)
0810 23:03:18.101934 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.101943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.101968 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:03:18.102069 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.102075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.102134 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.102301 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:18.103060 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.104549 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.105596 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.106085 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.106193 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.109042 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:18.109047 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:18.109047 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:18.109059 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:18.111032 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.111454 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.111457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.111457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.111579 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.111788 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.111789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.111789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.112085 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:18.112222 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.112348 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.112477 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260810 23:03:18.123174 25148 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:18.123883 25148 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:18.130551 25148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3: Generated new TSK 0
I20260810 23:03:18.130733 25148 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:18.160010 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45329
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:18.244567 25154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:18.244832 25154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:18.244877 25154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:18.246091 25154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:18.246156 25154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:18.246189 25154 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:03:18.246227 25154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:18.247814 25154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.56.126:45329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:18.248090 25154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:18.248327 25154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:18.253487 25160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:18.253531 25154 server_base.cc:1034] running on GCE node
W20260810 23:03:18.253480 25162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:18.253480 25159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:18.253891 25154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:18.254102 25154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:18.255230 25154 hybrid_clock.cc:648] HybridClock initialized: now 1786402998255209 us; error 40 us; skew 500 ppm
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:18.257750 25154 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:18.258186 25154 webserver.cc:460] Webserver started at http://127.23.56.65:42713/ using document root <none> and password file <none>
I20260810 23:03:18.258411 25154 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:18.258477 25154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:18.258630 25154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:18.259769 25154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "ce35ee3b9f29449ba869e5477c99b753"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "f4d4c741b6e3dd0b96b81495e9e0b942"
server_key_iv: "c01cf35b4351d2d4cd2f01ef36624f5e"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.260133 25154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "ce35ee3b9f29449ba869e5477c99b753"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "f4d4c741b6e3dd0b96b81495e9e0b942"
server_key_iv: "c01cf35b4351d2d4cd2f01ef36624f5e"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.261505 25154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:18.262241 25169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.262421 25154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:18.262527 25154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "ce35ee3b9f29449ba869e5477c99b753"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "f4d4c741b6e3dd0b96b81495e9e0b942"
server_key_iv: "c01cf35b4351d2d4cd2f01ef36624f5e"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.262599 25154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:18.273659 25154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:18.275547 25154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:18.275794 25154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:18.276319 25154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:18.276376 25154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.276427 25154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:18.276473 25154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.283052 25154 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:45731
I20260810 23:03:18.283205 25304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:45731 every 8 connection(s)
I20260810 23:03:18.283576 25154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:18.285799 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 25154
I20260810 23:03:18.285913 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:18.286085 23777 external_mini_cluster.cc:1428] Setting key defeed6b9cc9f721bc923ebfc3ca9368
I20260810 23:03:18.287338 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.276923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:58069 (local address 127.23.56.126:45329)
0810 23:03:18.276966 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.276968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.277245 (+ 277us) server_negotiation.cc:407] Connection header received
0810 23:03:18.277848 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.277850 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.277870 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.277944 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:18.278865 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.279761 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.281370 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.281722 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.281807 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.284583 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:18.284584 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:18.284585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:18.284597 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:18.285791 (+ 1194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.286251 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.286253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.286253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.286310 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.286583 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.286584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.286585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.286757 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:18.286838 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.287139 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.287241 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:03:18.287583 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.276947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45329 (local address 127.23.56.65:58069)
0810 23:03:18.277130 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.277165 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.277694 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.278044 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.278049 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.278447 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:18.278749 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.278754 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.279863 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.279865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.281268 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.281272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.281824 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.281825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.281979 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.282582 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:18.282592 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:18.284476 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:18.285878 (+ 1402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.285886 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.285888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.286181 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.286383 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.286384 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.286384 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.286521 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.286840 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:18.286849 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:18.286984 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.287096 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.287287 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260810 23:03:18.288774 25305 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45329
I20260810 23:03:18.288861 25305 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:18.289041 25305 heartbeater.cc:511] Master 127.23.56.126:45329 requested a full tablet report, sending...
I20260810 23:03:18.289666 25069 ts_manager.cc:194] Registered new tserver with Master: ce35ee3b9f29449ba869e5477c99b753 (127.23.56.65:45731)
I20260810 23:03:18.290303 25069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:58069
I20260810 23:03:18.295378 25312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.289137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45329 (local address 127.23.56.65:59917)
0810 23:03:18.289275 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.289286 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.289333 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.289482 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.289483 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.289704 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:18.289780 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.289782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.290467 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.290468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.291295 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.291298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.291681 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.291682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.291763 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.292229 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:18.292234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:18.292540 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:18.294373 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.294374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.294374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.294558 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.294766 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.294767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.294767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.294818 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.295078 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:18.295079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:18.295171 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.295236 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.295310 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260810 23:03:18.295492 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.289311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:59917 (local address 127.23.56.126:45329)
0810 23:03:18.289354 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.289356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.289380 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:03:18.289402 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.289403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.289414 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.289470 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:18.289865 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.290363 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.291384 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.291595 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.291678 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.292645 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:18.292647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:18.292647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:18.292654 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:18.294297 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.294632 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.294634 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.294634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.294698 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.294876 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.294877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.294877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.295017 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:18.295089 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.295284 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.295375 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:18.314504 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45329
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:18.395771 25314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:18.395958 25314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:18.395982 25314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:18.397116 25314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:18.397161 25314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:18.397181 25314 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:03:18.397198 25314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:18.398614 25314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.56.126:45329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:18.398849 25314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:18.399075 25314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:18.403904 25319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:18.403926 25323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:18.403932 25320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:18.404265 25314 server_base.cc:1034] running on GCE node
I20260810 23:03:18.404415 25314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:18.404693 25314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:18.405848 25314 hybrid_clock.cc:648] HybridClock initialized: now 1786402998405830 us; error 29 us; skew 500 ppm
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:18.408282 25314 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:18.408733 25314 webserver.cc:460] Webserver started at http://127.23.56.66:38767/ using document root <none> and password file <none>
I20260810 23:03:18.408998 25314 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:18.409063 25314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:18.409200 25314 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:18.410180 25314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "7cf7f60daae34291af3a2f217ca4d4c2"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "a6530eba903ba521013641ca030e0d38"
server_key_iv: "19ac54d3be3c64a301e3860468ab0aac"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.410522 25314 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "7cf7f60daae34291af3a2f217ca4d4c2"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "a6530eba903ba521013641ca030e0d38"
server_key_iv: "19ac54d3be3c64a301e3860468ab0aac"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.411870 25314 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:18.412532 25330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.412694 25314 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.412789 25314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "7cf7f60daae34291af3a2f217ca4d4c2"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "a6530eba903ba521013641ca030e0d38"
server_key_iv: "19ac54d3be3c64a301e3860468ab0aac"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.412855 25314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:18.424417 25314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:18.426507 25314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:18.426657 25314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:18.427203 25314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:18.427256 25314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.427325 25314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:18.427366 25314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.434751 25314 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:32991
I20260810 23:03:18.434813 25460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:32991 every 8 connection(s)
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:18.435268 25314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:18.438083 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.427538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:49287 (local address 127.23.56.126:45329)
0810 23:03:18.427600 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.427603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.427822 (+ 219us) server_negotiation.cc:407] Connection header received
0810 23:03:18.428523 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.428524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.428535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.428577 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:18.429430 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.429928 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.431611 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.431832 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.431914 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.435657 (+ 3743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:18.435659 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:18.435660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:18.435668 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:18.436822 (+ 1154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.437322 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.437323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.437323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.437375 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.437583 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.437583 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.437583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.437686 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:18.437734 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.437960 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.438021 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 23:03:18.438325 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.427525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45329 (local address 127.23.56.66:49287)
0810 23:03:18.427682 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.427730 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.428397 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.428657 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.428661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.429029 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:18.429327 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.429334 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.429998 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.429999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.431518 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.431521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.431935 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.431937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.432063 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.432544 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:18.432553 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:18.435564 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:18.436889 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.436895 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.436896 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.437250 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.437432 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.437432 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.437433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.437531 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.437740 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:18.437745 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:18.437866 (+ 121us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.437948 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.438065 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260810 23:03:18.438870 25461 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45329
I20260810 23:03:18.438969 25461 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:18.439113 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 25314
I20260810 23:03:18.439203 25461 heartbeater.cc:511] Master 127.23.56.126:45329 requested a full tablet report, sending...
I20260810 23:03:18.439227 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:18.439453 23777 external_mini_cluster.cc:1428] Setting key 8c792490ba118f0b2b1c6be029242712
I20260810 23:03:18.439733 25069 ts_manager.cc:194] Registered new tserver with Master: 7cf7f60daae34291af3a2f217ca4d4c2 (127.23.56.66:32991)
I20260810 23:03:18.440207 25069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:49287
I20260810 23:03:18.446126 25468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.440027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45329 (local address 127.23.56.66:48293)
0810 23:03:18.440150 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.440160 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.440212 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.440385 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.440386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.440598 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:18.440672 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.440674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.441557 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.441557 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.442027 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.442028 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.442400 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.442401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.442475 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.442912 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:18.442917 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:18.443271 (+ 354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:18.445020 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.445021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.445021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.445251 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.445474 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.445475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.445475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.445545 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.445889 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:18.445890 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:18.445924 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.445983 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.446058 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260810 23:03:18.446230 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.440152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:48293 (local address 127.23.56.126:45329)
0810 23:03:18.440198 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.440200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.440209 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:18.440279 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.440280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.440291 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.440323 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:18.440759 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.441463 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.442111 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.442316 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.442397 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.443369 (+ 972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:18.443371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:18.443371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:18.443380 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:18.444936 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.445323 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.445324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.445325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.445389 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.445615 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.445616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.445617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.445825 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:18.445900 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.446020 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.446103 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:18.466142 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45329
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:18.548012 25472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:18.548230 25472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:18.548273 25472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:18.549409 25472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:18.549471 25472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:18.549506 25472 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:03:18.549547 25472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:18.550944 25472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.56.126:45329
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:18.551235 25472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:18.551457 25472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:18.556317 25480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:18.556324 25477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:18.556567 25472 server_base.cc:1034] running on GCE node
W20260810 23:03:18.556317 25478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:18.556830 25472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:18.557085 25472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:18.558254 25472 hybrid_clock.cc:648] HybridClock initialized: now 1786402998558243 us; error 35 us; skew 500 ppm
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:18.560717 25472 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:18.561249 25472 webserver.cc:460] Webserver started at http://127.23.56.67:45419/ using document root <none> and password file <none>
I20260810 23:03:18.561465 25472 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:18.561528 25472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:18.561666 25472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:18.562638 25472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "8efa24a8ab33453fab13611bb6281948"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "be89ae2f532c235c5b3240ecb5469488"
server_key_iv: "7adc4321cb2a31f5ab18a4852373b29d"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.563064 25472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "8efa24a8ab33453fab13611bb6281948"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "be89ae2f532c235c5b3240ecb5469488"
server_key_iv: "7adc4321cb2a31f5ab18a4852373b29d"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.564435 25472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:18.565307 25488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.565502 25472 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:18.565584 25472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "8efa24a8ab33453fab13611bb6281948"
format_stamp: "Formatted at 2026-08-10 23:03:18 on dist-test-slave-1xrh"
server_key: "be89ae2f532c235c5b3240ecb5469488"
server_key_iv: "7adc4321cb2a31f5ab18a4852373b29d"
server_key_version: "encryptionkey@0"
I20260810 23:03:18.565665 25472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:18.579783 25472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:18.581530 25472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:18.581655 25472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:18.582123 25472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:18.582165 25472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:18.582185 25472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:18.582224 25472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:18.589484 25472 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:42633
I20260810 23:03:18.589560 25613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:42633 every 8 connection(s)
I20260810 23:03:18.589890 25472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:18.590982 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 25472
I20260810 23:03:18.591058 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:18.591253 23777 external_mini_cluster.cc:1428] Setting key 94a384057906097671186ac69f6cbea2
I20260810 23:03:18.592705 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.582615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:33349 (local address 127.23.56.126:45329)
0810 23:03:18.582685 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.582688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.582849 (+ 161us) server_negotiation.cc:407] Connection header received
0810 23:03:18.583411 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.583413 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.583426 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.583521 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:18.584348 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.585180 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.586686 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.587015 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.587084 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.589790 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:18.589791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:18.589791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:18.589798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:18.591397 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.591871 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.591873 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.591873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.591937 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.592218 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.592219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.592219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.592341 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:18.592391 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.592584 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.592643 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:03:18.592864 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.582573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45329 (local address 127.23.56.67:33349)
0810 23:03:18.582723 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.582761 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.583310 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.583605 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.583609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.583967 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:18.584242 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.584249 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.585270 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.585271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.586588 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.586591 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.587111 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.587112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.587264 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.587720 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:18.587734 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:18.589704 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:18.591457 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.591463 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.591466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.591805 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.592004 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.592005 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.592006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.592139 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.592384 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:18.592388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:18.592494 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.592570 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.592680 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260810 23:03:18.593289 25614 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45329
I20260810 23:03:18.593372 25614 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:18.593555 25614 heartbeater.cc:511] Master 127.23.56.126:45329 requested a full tablet report, sending...
I20260810 23:03:18.593910 25069 ts_manager.cc:194] Registered new tserver with Master: 8efa24a8ab33453fab13611bb6281948 (127.23.56.67:42633)
I20260810 23:03:18.594296 25069 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:33349
I20260810 23:03:18.599890 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.594367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:50693 (local address 127.23.56.126:45329)
0810 23:03:18.594406 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.594408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.594526 (+ 118us) server_negotiation.cc:407] Connection header received
0810 23:03:18.594973 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.594974 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.594983 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.595057 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:18.595417 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.595864 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.596579 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.596786 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.596863 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.597733 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:18.597735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:18.597735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:18.597742 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:18.598804 (+ 1062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.599153 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.599154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.599154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.599210 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.599392 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.599393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.599393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.599496 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:18.599562 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.599744 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.599809 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:03:18.599906 25620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.594284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45329 (local address 127.23.56.67:50693)
0810 23:03:18.594460 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.594471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.594531 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.595081 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.595082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.595269 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:18.595329 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.595330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.595964 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.595965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.596479 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.596480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.596866 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.596867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.596938 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.597334 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:18.597339 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:18.597624 (+ 285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:18.598892 (+ 1268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.598892 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.598893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.599053 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.599272 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:18.599272 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:18.599273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:18.599327 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:18.599557 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:18.599558 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:18.599621 (+ 63us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.599704 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.599806 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":77,"threads_started":1}
I20260810 23:03:18.602869 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:03:18 dist-test-slave-1xrh krb5kdc[25018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:18.625164 25130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.618715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44824 (local address 127.23.56.126:45329)
0810 23:03:18.618769 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.618771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.618779 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:03:18.618825 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.618826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.618841 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.618894 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:18.619476 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.619942 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.620636 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.620830 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.620895 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.622891 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:18.622893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:18.622893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:18.622900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:18.624051 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.624387 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.624388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.624388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.624443 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:18.624649 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:18.624650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:18.624651 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:18.624756 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:18.624856 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.624975 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.625061 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 23:03:18.626765 25069 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:18.628823 25069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:18.635181 25628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.630933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:32991 (local address 127.0.0.1:46526)
0810 23:03:18.631077 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.631110 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.631202 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.631897 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.631900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.631902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:18.631987 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.631990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.632865 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.632867 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.634021 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.634024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.634851 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.634852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.634942 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.634956 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.635037 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.635085 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":4,"thread_start_us":80,"threads_started":1}
I20260810 23:03:18.635274 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.631236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46526 (local address 127.23.56.66:32991)
0810 23:03:18.631725 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.631729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.631738 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:18.631773 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.631776 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.631790 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.632068 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:18.632125 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.632777 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.634095 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.634756 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.634834 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.634889 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.635062 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.635107 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":93,"threads_started":1}
I20260810 23:03:18.635596 25631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.631189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:45731 (local address 127.0.0.1:52934)
0810 23:03:18.631424 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.631435 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.631496 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.631902 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.631903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.631906 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:18.632001 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.632004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.633277 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.633278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.634289 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.634291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.635346 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.635347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.635422 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.635427 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.635483 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.635523 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":23,"thread_start_us":52,"threads_started":1}
I20260810 23:03:18.635664 25632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.631316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52934 (local address 127.23.56.65:45731)
0810 23:03:18.631724 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.631729 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.631740 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:03:18.631779 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.631781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.631797 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.631944 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:18.632087 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.633178 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.634362 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.635104 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.635247 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.635315 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.635507 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.635561 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":77,"threads_started":1}
I20260810 23:03:18.636119 25385 tablet_service.cc:1467] Processing CreateTablet for tablet 445275d0c7a84bf98bd0a0e24907ab7a (DEFAULT_TABLE table=test-table [id=2a5c9306083e489495d49e1c71e98c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:18.636391 25629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.631335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:42633 (local address 127.0.0.1:57084)
0810 23:03:18.631558 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.631564 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.631590 (+ 26us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.631897 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.631900 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.631905 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:18.631987 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.631999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.634357 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.634357 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.635195 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.635197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.636168 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.636169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.636234 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.636237 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.636276 (+ 39us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.636305 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":9,"thread_start_us":45,"threads_started":1}
I20260810 23:03:18.636422 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.631508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.23.56.67:42633)
0810 23:03:18.631670 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.631676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.631688 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:03:18.631727 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.631729 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.631748 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.631832 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:18.632062 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.633178 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.635291 (+ 2113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.636082 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.636168 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.636225 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.636290 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.636334 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":89,"threads_started":1}
I20260810 23:03:18.636533 25385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445275d0c7a84bf98bd0a0e24907ab7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:18.639783 25542 tablet_service.cc:1467] Processing CreateTablet for tablet 445275d0c7a84bf98bd0a0e24907ab7a (DEFAULT_TABLE table=test-table [id=2a5c9306083e489495d49e1c71e98c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:18.639786 25228 tablet_service.cc:1467] Processing CreateTablet for tablet 445275d0c7a84bf98bd0a0e24907ab7a (DEFAULT_TABLE table=test-table [id=2a5c9306083e489495d49e1c71e98c2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:18.640138 25228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445275d0c7a84bf98bd0a0e24907ab7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:18.640139 25542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 445275d0c7a84bf98bd0a0e24907ab7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:18.643270 25635 tablet_bootstrap.cc:492] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: Bootstrap starting.
I20260810 23:03:18.643422 25634 tablet_bootstrap.cc:492] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: Bootstrap starting.
I20260810 23:03:18.643563 25636 tablet_bootstrap.cc:492] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: Bootstrap starting.
I20260810 23:03:18.644115 25634 tablet_bootstrap.cc:654] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:18.644351 25636 tablet_bootstrap.cc:654] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:18.644351 25635 tablet_bootstrap.cc:654] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:18.644483 25634 log.cc:824] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:18.644704 25636 log.cc:824] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:18.644704 25635 log.cc:824] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:18.645252 25634 tablet_bootstrap.cc:492] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: No bootstrap required, opened a new log
I20260810 23:03:18.645371 25634 ts_tablet_manager.cc:1397] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:18.645416 25636 tablet_bootstrap.cc:492] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: No bootstrap required, opened a new log
I20260810 23:03:18.645474 25636 ts_tablet_manager.cc:1397] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:03:18.645661 25635 tablet_bootstrap.cc:492] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: No bootstrap required, opened a new log
I20260810 23:03:18.645725 25635 ts_tablet_manager.cc:1397] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260810 23:03:18.646963 25634 raft_consensus.cc:348] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.647109 25634 raft_consensus.cc:374] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:18.647197 25634 raft_consensus.cc:729] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf7f60daae34291af3a2f217ca4d4c2, State: Initialized, Role: FOLLOWER
I20260810 23:03:18.647323 25634 consensus_queue.cc:260] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.647454 25636 raft_consensus.cc:348] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.647601 25634 ts_tablet_manager.cc:1428] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:18.647678 25636 raft_consensus.cc:374] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:18.647665 25461 heartbeater.cc:503] Master 127.23.56.126:45329 was elected leader, sending a full tablet report...
I20260810 23:03:18.647609 25635 raft_consensus.cc:348] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.647729 25635 raft_consensus.cc:374] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:18.647773 25635 raft_consensus.cc:729] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efa24a8ab33453fab13611bb6281948, State: Initialized, Role: FOLLOWER
I20260810 23:03:18.647805 25636 raft_consensus.cc:729] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce35ee3b9f29449ba869e5477c99b753, State: Initialized, Role: FOLLOWER
I20260810 23:03:18.647883 25635 consensus_queue.cc:260] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.647964 25636 consensus_queue.cc:260] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.648172 25635 ts_tablet_manager.cc:1428] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:18.648206 25636 ts_tablet_manager.cc:1428] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:18.648245 25305 heartbeater.cc:503] Master 127.23.56.126:45329 was elected leader, sending a full tablet report...
I20260810 23:03:18.648414 25614 heartbeater.cc:503] Master 127.23.56.126:45329 was elected leader, sending a full tablet report...
W20260810 23:03:18.685901 25462 tablet.cc:2369] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:18.784368 25306 tablet.cc:2369] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:18.840553 25615 tablet.cc:2369] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:03:18.984808 25641 raft_consensus.cc:482] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:18.984938 25641 raft_consensus.cc:504] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.985328 25641 leader_election.cc:288] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cf7f60daae34291af3a2f217ca4d4c2 (127.23.56.66:32991), ce35ee3b9f29449ba869e5477c99b753 (127.23.56.65:45731)
I20260810 23:03:18.987978 25640 raft_consensus.cc:482] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:18.988106 25640 raft_consensus.cc:504] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.988469 25640 leader_election.cc:288] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce35ee3b9f29449ba869e5477c99b753 (127.23.56.65:45731), 8efa24a8ab33453fab13611bb6281948 (127.23.56.67:42633)
I20260810 23:03:18.989377 25632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.985716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:36773 (local address 127.23.56.65:45731)
0810 23:03:18.985777 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.985779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.985872 (+ 93us) server_negotiation.cc:407] Connection header received
0810 23:03:18.985954 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.985955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.985964 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.986019 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:18.986154 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.986862 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.988299 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.988885 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.988958 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.988967 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.989255 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.989303 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 23:03:18.989390 25497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.985523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:32991 (local address 127.23.56.67:48951)
0810 23:03:18.985582 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.985597 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.985652 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.985815 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.985817 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.985819 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:18.985899 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.985902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.986873 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.986874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.988222 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.988225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.989150 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.989151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.989237 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.989240 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.989287 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.989324 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":78}
I20260810 23:03:18.989634 25643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.985635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:45731 (local address 127.23.56.67:36773)
0810 23:03:18.985804 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.985815 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.985876 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.986039 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.986040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.986041 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:18.986093 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.986094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.986929 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.986930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.988222 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.988225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.989014 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.989015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.989166 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.989171 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.989492 (+ 321us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.989537 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":4,"thread_start_us":100,"threads_started":1}
I20260810 23:03:18.990062 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.985604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:48951 (local address 127.23.56.66:32991)
0810 23:03:18.985682 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.985684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.985693 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:18.985721 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.985722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.985731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.985778 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:18.985968 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.986806 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.988362 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.989050 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.989113 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.989122 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.989949 (+ 827us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.990000 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:03:18.990118 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445275d0c7a84bf98bd0a0e24907ab7a" candidate_uuid: "8efa24a8ab33453fab13611bb6281948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce35ee3b9f29449ba869e5477c99b753" is_pre_election: true
I20260810 23:03:18.990291 25250 raft_consensus.cc:2454] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8efa24a8ab33453fab13611bb6281948 in term 0.
I20260810 23:03:18.990306 25408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445275d0c7a84bf98bd0a0e24907ab7a" candidate_uuid: "8efa24a8ab33453fab13611bb6281948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" is_pre_election: true
I20260810 23:03:18.990406 25408 raft_consensus.cc:2454] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8efa24a8ab33453fab13611bb6281948 in term 0.
I20260810 23:03:18.990622 25489 leader_election.cc:302] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cf7f60daae34291af3a2f217ca4d4c2, 8efa24a8ab33453fab13611bb6281948; no voters:
I20260810 23:03:18.990734 25641 raft_consensus.cc:2781] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:18.990778 25641 raft_consensus.cc:482] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:18.990805 25641 raft_consensus.cc:3037] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:18.992081 25641 raft_consensus.cc:504] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.992414 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.988981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:39843 (local address 127.23.56.67:42633)
0810 23:03:18.989018 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.989020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.989144 (+ 124us) server_negotiation.cc:407] Connection header received
0810 23:03:18.989237 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.989238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.989246 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.989275 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:18.989461 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.990107 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.991504 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.992036 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.992113 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.992122 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.992315 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.992359 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:03:18.992388 25408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445275d0c7a84bf98bd0a0e24907ab7a" candidate_uuid: "8efa24a8ab33453fab13611bb6281948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf7f60daae34291af3a2f217ca4d4c2"
I20260810 23:03:18.992457 25644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.988834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:42633 (local address 127.23.56.66:39843)
0810 23:03:18.989090 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.989099 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.989166 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.989330 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.989331 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.989332 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:18.989388 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.989389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.990210 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.990211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.991407 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.991410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.992157 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.992159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.992233 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.992236 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.992298 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.992343 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":54,"threads_started":1}
I20260810 23:03:18.992462 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.988721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:57003 (local address 127.23.56.65:45731)
0810 23:03:18.988818 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:18.988819 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:18.988999 (+ 180us) server_negotiation.cc:407] Connection header received
0810 23:03:18.989111 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:18.989112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:18.989123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:18.989164 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:18.989394 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.990542 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.991728 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.992090 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.992169 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.992177 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:18.992347 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 23:03:18.992385 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
I20260810 23:03:18.992472 25408 raft_consensus.cc:3037] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:18.992738 25646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:18.988678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:45731 (local address 127.23.56.66:57003)
0810 23:03:18.988928 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:18.988939 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:18.989030 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:18.989242 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:18.989243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:18.989245 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:18.989311 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.989313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.990635 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.990636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.991647 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:18.991649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:18.992168 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:18.992169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:18.992246 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:18.992249 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:18.992564 (+ 315us) client_negotiation.cc:240] Negotiation successful
0810 23:03:18.992618 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":57,"threads_started":1}
I20260810 23:03:18.993047 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445275d0c7a84bf98bd0a0e24907ab7a" candidate_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce35ee3b9f29449ba869e5477c99b753" is_pre_election: true
I20260810 23:03:18.993130 25250 raft_consensus.cc:2454] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7cf7f60daae34291af3a2f217ca4d4c2 in term 0.
I20260810 23:03:18.993101 25566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445275d0c7a84bf98bd0a0e24907ab7a" candidate_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efa24a8ab33453fab13611bb6281948" is_pre_election: true
I20260810 23:03:18.993227 25566 raft_consensus.cc:2379] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7cf7f60daae34291af3a2f217ca4d4c2 in current term 1: Already voted for candidate 8efa24a8ab33453fab13611bb6281948 in this term.
I20260810 23:03:18.993314 25331 leader_election.cc:302] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cf7f60daae34291af3a2f217ca4d4c2, ce35ee3b9f29449ba869e5477c99b753; no voters:
I20260810 23:03:18.993455 25408 raft_consensus.cc:2454] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8efa24a8ab33453fab13611bb6281948 in term 1.
I20260810 23:03:18.993479 25641 leader_election.cc:288] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [CANDIDATE]: Term 1 election: Requested vote from peers 7cf7f60daae34291af3a2f217ca4d4c2 (127.23.56.66:32991), ce35ee3b9f29449ba869e5477c99b753 (127.23.56.65:45731)
I20260810 23:03:18.993548 25640 raft_consensus.cc:2741] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 23:03:18.993613 25489 leader_election.cc:302] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cf7f60daae34291af3a2f217ca4d4c2, 8efa24a8ab33453fab13611bb6281948; no voters:
I20260810 23:03:18.993631 25250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "445275d0c7a84bf98bd0a0e24907ab7a" candidate_uuid: "8efa24a8ab33453fab13611bb6281948" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce35ee3b9f29449ba869e5477c99b753"
I20260810 23:03:18.993695 25250 raft_consensus.cc:3037] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:18.993712 25641 raft_consensus.cc:2781] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:18.993870 25641 raft_consensus.cc:686] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 1 LEADER]: Becoming Leader. State: Replica: 8efa24a8ab33453fab13611bb6281948, State: Running, Role: LEADER
I20260810 23:03:18.993970 25641 consensus_queue.cc:237] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:18.994625 25250 raft_consensus.cc:2454] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8efa24a8ab33453fab13611bb6281948 in term 1.
I20260810 23:03:18.995196 25070 catalog_manager.cc:5495] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8efa24a8ab33453fab13611bb6281948 (127.23.56.67). New cstate: current_term: 1 leader_uuid: "8efa24a8ab33453fab13611bb6281948" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } health_report { overall_health: HEALTHY } } }
I20260810 23:03:19.073884 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.071613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57090 (local address 127.23.56.67:42633)
0810 23:03:19.071668 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:19.071670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:19.071686 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:03:19.071775 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:19.071776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:19.071796 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:19.071848 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:19.071996 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.072476 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.073206 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.073395 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.073443 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.073606 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:19.073693 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:19.073747 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:19.073794 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 23:03:19.073810 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 23:03:19.075685 25408 raft_consensus.cc:1261] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Refusing update from remote peer 8efa24a8ab33453fab13611bb6281948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:19.075685 25250 raft_consensus.cc:1261] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 1 FOLLOWER]: Refusing update from remote peer 8efa24a8ab33453fab13611bb6281948: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:19.075999 25641 consensus_queue.cc:1035] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:19.076125 25647 consensus_queue.cc:1035] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:19.078436 25651 mvcc.cc:205] Tried to move back new op lower bound from 7317106684210896896 to 7317106683879927808. Current Snapshot: MvccSnapshot[applied={T|T < 7317106684210896896}]
I20260810 23:03:19.081645 25069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:19.081725 25069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:19.082433 25069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3: Sending DeleteTablet for 3 replicas of tablet 445275d0c7a84bf98bd0a0e24907ab7a
I20260810 23:03:19.082741 25385 tablet_service.cc:1514] Processing DeleteTablet for tablet 445275d0c7a84bf98bd0a0e24907ab7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:19 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:46526
I20260810 23:03:19.082834 25228 tablet_service.cc:1514] Processing DeleteTablet for tablet 445275d0c7a84bf98bd0a0e24907ab7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:19 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:52934
I20260810 23:03:19.083006 25542 tablet_service.cc:1514] Processing DeleteTablet for tablet 445275d0c7a84bf98bd0a0e24907ab7a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:19 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:57084
I20260810 23:03:19.083026 25069 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:19.083180 25069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:19.083369 25662 tablet_replica.cc:330] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: stopping tablet replica
I20260810 23:03:19.083496 25660 tablet_replica.cc:330] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: stopping tablet replica
I20260810 23:03:19.083485 25661 tablet_replica.cc:330] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: stopping tablet replica
I20260810 23:03:19.083592 25661 raft_consensus.cc:2229] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:19.083596 25660 raft_consensus.cc:2229] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:19.083695 25660 raft_consensus.cc:2258] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:19.083719 25661 raft_consensus.cc:2258] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:19.083854 25662 raft_consensus.cc:2229] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:19.083974 25662 raft_consensus.cc:2258] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:19.084331 25660 ts_tablet_manager.cc:1905] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:19.084388 25662 ts_tablet_manager.cc:1905] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:19.084506 25661 ts_tablet_manager.cc:1905] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:19.085042 25385 tablet_service.cc:1467] Processing CreateTablet for tablet 90e9be3d51da439d9278ec2208be7568 (DEFAULT_TABLE table=test-table [id=b73c8d1f6c0a403f86ca9208248e1c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:19.085161 25385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e9be3d51da439d9278ec2208be7568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:19.086324 25662 ts_tablet_manager.cc:1918] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:19.086390 25662 log.cc:1196] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/445275d0c7a84bf98bd0a0e24907ab7a
I20260810 23:03:19.086655 25662 ts_tablet_manager.cc:1939] T 445275d0c7a84bf98bd0a0e24907ab7a P 8efa24a8ab33453fab13611bb6281948: Deleting consensus metadata
I20260810 23:03:19.086686 25634 tablet_bootstrap.cc:492] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: Bootstrap starting.
I20260810 23:03:19.087110 25634 tablet_bootstrap.cc:654] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:19.087175 25542 tablet_service.cc:1467] Processing CreateTablet for tablet 90e9be3d51da439d9278ec2208be7568 (DEFAULT_TABLE table=test-table [id=b73c8d1f6c0a403f86ca9208248e1c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:19.087275 25542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e9be3d51da439d9278ec2208be7568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:19.087322 25052 catalog_manager.cc:4841] TS 8efa24a8ab33453fab13611bb6281948 (127.23.56.67:42633): tablet 445275d0c7a84bf98bd0a0e24907ab7a (table test-table [id=2a5c9306083e489495d49e1c71e98c2a]) successfully deleted
I20260810 23:03:19.087520 25660 ts_tablet_manager.cc:1918] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:19.087607 25660 log.cc:1196] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/445275d0c7a84bf98bd0a0e24907ab7a
I20260810 23:03:19.087809 25634 tablet_bootstrap.cc:492] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: No bootstrap required, opened a new log
I20260810 23:03:19.087862 25634 ts_tablet_manager.cc:1397] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:19.087940 25660 ts_tablet_manager.cc:1939] T 445275d0c7a84bf98bd0a0e24907ab7a P 7cf7f60daae34291af3a2f217ca4d4c2: Deleting consensus metadata
I20260810 23:03:19.088040 25634 raft_consensus.cc:348] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.088105 25634 raft_consensus.cc:374] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:19.088124 25634 raft_consensus.cc:729] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf7f60daae34291af3a2f217ca4d4c2, State: Initialized, Role: FOLLOWER
I20260810 23:03:19.084893 25228 tablet_service.cc:1467] Processing CreateTablet for tablet 90e9be3d51da439d9278ec2208be7568 (DEFAULT_TABLE table=test-table [id=b73c8d1f6c0a403f86ca9208248e1c73]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:19.088191 25634 consensus_queue.cc:260] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.088280 25228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e9be3d51da439d9278ec2208be7568. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:19.088289 25634 ts_tablet_manager.cc:1428] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:19.088471 25635 tablet_bootstrap.cc:492] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: Bootstrap starting.
I20260810 23:03:19.088570 25052 catalog_manager.cc:4841] TS 7cf7f60daae34291af3a2f217ca4d4c2 (127.23.56.66:32991): tablet 445275d0c7a84bf98bd0a0e24907ab7a (table test-table [id=2a5c9306083e489495d49e1c71e98c2a]) successfully deleted
I20260810 23:03:19.088833 25635 tablet_bootstrap.cc:654] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:19.089372 25635 tablet_bootstrap.cc:492] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: No bootstrap required, opened a new log
I20260810 23:03:19.089411 25635 ts_tablet_manager.cc:1397] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260810 23:03:19.089570 25635 raft_consensus.cc:348] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.089637 25635 raft_consensus.cc:374] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:19.089668 25635 raft_consensus.cc:729] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efa24a8ab33453fab13611bb6281948, State: Initialized, Role: FOLLOWER
I20260810 23:03:19.089694 25636 tablet_bootstrap.cc:492] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: Bootstrap starting.
I20260810 23:03:19.089711 25635 consensus_queue.cc:260] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.089797 25635 ts_tablet_manager.cc:1428] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:19.090104 25636 tablet_bootstrap.cc:654] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:19.090700 25636 tablet_bootstrap.cc:492] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: No bootstrap required, opened a new log
I20260810 23:03:19.090761 25636 ts_tablet_manager.cc:1397] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:19.090901 25636 raft_consensus.cc:348] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.090945 25636 raft_consensus.cc:374] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:19.090957 25636 raft_consensus.cc:729] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce35ee3b9f29449ba869e5477c99b753, State: Initialized, Role: FOLLOWER
I20260810 23:03:19.091014 25636 consensus_queue.cc:260] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.091089 25636 ts_tablet_manager.cc:1428] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:19.091661 25661 ts_tablet_manager.cc:1918] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:19.091754 25661 log.cc:1196] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/445275d0c7a84bf98bd0a0e24907ab7a
I20260810 23:03:19.092000 25661 ts_tablet_manager.cc:1939] T 445275d0c7a84bf98bd0a0e24907ab7a P ce35ee3b9f29449ba869e5477c99b753: Deleting consensus metadata
I20260810 23:03:19.092352 25052 catalog_manager.cc:4841] TS ce35ee3b9f29449ba869e5477c99b753 (127.23.56.65:45731): tablet 445275d0c7a84bf98bd0a0e24907ab7a (table test-table [id=2a5c9306083e489495d49e1c71e98c2a]) successfully deleted
I20260810 23:03:19.172751 25642 raft_consensus.cc:482] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:19.172871 25642 raft_consensus.cc:504] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.173218 25642 leader_election.cc:288] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cf7f60daae34291af3a2f217ca4d4c2 (127.23.56.66:32991), 8efa24a8ab33453fab13611bb6281948 (127.23.56.67:42633)
I20260810 23:03:19.176981 25630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.173560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:34799 (local address 127.23.56.66:32991)
0810 23:03:19.173634 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:19.173636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:19.173710 (+ 74us) server_negotiation.cc:407] Connection header received
0810 23:03:19.173836 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:19.173838 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:19.173848 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:19.173879 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:19.174202 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.174995 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.176181 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.176591 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.176689 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.176698 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:19.176866 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 23:03:19.176905 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:03:19.176996 25666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.173430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:32991 (local address 127.23.56.65:34799)
0810 23:03:19.173636 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:19.173648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:19.173745 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:19.173970 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:19.173971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:19.173976 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:19.174092 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.174095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.175093 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.175094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.176097 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.176101 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.176721 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.176723 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.176793 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.176796 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:19.176859 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:03:19.176900 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":5,"thread_start_us":72,"threads_started":1}
I20260810 23:03:19.177261 25408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e9be3d51da439d9278ec2208be7568" candidate_uuid: "ce35ee3b9f29449ba869e5477c99b753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" is_pre_election: true
I20260810 23:03:19.177361 25408 raft_consensus.cc:2454] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce35ee3b9f29449ba869e5477c99b753 in term 0.
I20260810 23:03:19.177600 25170 leader_election.cc:302] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cf7f60daae34291af3a2f217ca4d4c2, ce35ee3b9f29449ba869e5477c99b753; no voters:
I20260810 23:03:19.177726 25642 raft_consensus.cc:2781] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:19.177757 25642 raft_consensus.cc:482] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:19.177779 25642 raft_consensus.cc:3037] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:19.178309 25667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.173615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:42633 (local address 127.23.56.65:36453)
0810 23:03:19.173751 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:19.173760 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:19.173801 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:19.173975 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:19.173975 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:19.173976 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:19.174092 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.174095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.176023 (+ 1928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.176024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.177323 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.177327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.178060 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.178060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.178124 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.178128 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:19.178186 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 23:03:19.178221 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":21,"thread_start_us":76,"threads_started":1}
I20260810 23:03:19.178334 25633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.173752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:36453 (local address 127.23.56.67:42633)
0810 23:03:19.173795 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:19.173797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:19.173807 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:19.173868 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:19.173869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:19.173878 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:19.173953 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:19.175112 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.175900 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.177427 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.177974 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.178049 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.178058 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:19.178211 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:03:19.178251 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:03:19.178757 25642 raft_consensus.cc:504] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.178896 25642 leader_election.cc:288] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [CANDIDATE]: Term 1 election: Requested vote from peers 7cf7f60daae34291af3a2f217ca4d4c2 (127.23.56.66:32991), 8efa24a8ab33453fab13611bb6281948 (127.23.56.67:42633)
I20260810 23:03:19.178906 25566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e9be3d51da439d9278ec2208be7568" candidate_uuid: "ce35ee3b9f29449ba869e5477c99b753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efa24a8ab33453fab13611bb6281948" is_pre_election: true
I20260810 23:03:19.179013 25566 raft_consensus.cc:2454] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce35ee3b9f29449ba869e5477c99b753 in term 0.
I20260810 23:03:19.179332 25408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e9be3d51da439d9278ec2208be7568" candidate_uuid: "ce35ee3b9f29449ba869e5477c99b753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf7f60daae34291af3a2f217ca4d4c2"
I20260810 23:03:19.179379 25566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e9be3d51da439d9278ec2208be7568" candidate_uuid: "ce35ee3b9f29449ba869e5477c99b753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efa24a8ab33453fab13611bb6281948"
I20260810 23:03:19.179426 25408 raft_consensus.cc:3037] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:19.179448 25566 raft_consensus.cc:3037] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:19.180016 25408 raft_consensus.cc:2454] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce35ee3b9f29449ba869e5477c99b753 in term 1.
I20260810 23:03:19.180176 25170 leader_election.cc:302] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7cf7f60daae34291af3a2f217ca4d4c2, ce35ee3b9f29449ba869e5477c99b753; no voters:
I20260810 23:03:19.180279 25642 raft_consensus.cc:2781] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:19.180447 25642 raft_consensus.cc:686] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 1 LEADER]: Becoming Leader. State: Replica: ce35ee3b9f29449ba869e5477c99b753, State: Running, Role: LEADER
I20260810 23:03:19.180517 25642 consensus_queue.cc:237] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.180627 25566 raft_consensus.cc:2454] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce35ee3b9f29449ba869e5477c99b753 in term 1.
I20260810 23:03:19.181144 25069 catalog_manager.cc:5495] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce35ee3b9f29449ba869e5477c99b753 (127.23.56.65). New cstate: current_term: 1 leader_uuid: "ce35ee3b9f29449ba869e5477c99b753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:03:19 dist-test-slave-1xrh krb5kdc[25018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:19.230796 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.223765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45329 (local address 127.0.0.1:44832)
0810 23:03:19.223974 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:19.223987 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:19.224066 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:19.224235 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:19.224236 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:19.224557 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:19.224603 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.224605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.225279 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.225280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.225796 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.225798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.226159 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.226161 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.226223 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.226732 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:19.226737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:19.228162 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:19.229686 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:19.229694 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:19.229697 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:19.229882 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:19.230081 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:19.230082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:19.230082 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:19.230140 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:19.230406 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:19.230413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:19.230458 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 23:03:19.230537 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:03:19.230672 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":122,"threads_started":1}
I20260810 23:03:19.230810 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.223854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.23.56.126:45329)
0810 23:03:19.224007 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:19.224009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:19.224042 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:03:19.224136 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:19.224137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:19.224149 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:19.224232 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:19.224692 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.225192 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.225885 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.226080 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.226153 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.228250 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:19.228252 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:19.228252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:19.228259 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:19.229598 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:19.229953 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:19.229954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:19.229954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:19.230018 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:19.230199 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:19.230200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:19.230200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:19.230340 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:19.230413 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:19.230571 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 23:03:19.230727 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":82,"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260810 23:03:19.233003 25069 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:44832:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:03:19.233217 25069 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:19.234622 25228 tablet_service.cc:1467] Processing CreateTablet for tablet 7646288466ab42d997f16c8b73303e9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a27ec0cd79264ec2911c24d690ee0303]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:03:19.234630 25385 tablet_service.cc:1467] Processing CreateTablet for tablet 7646288466ab42d997f16c8b73303e9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a27ec0cd79264ec2911c24d690ee0303]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:03:19.234642 25542 tablet_service.cc:1467] Processing CreateTablet for tablet 7646288466ab42d997f16c8b73303e9d (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a27ec0cd79264ec2911c24d690ee0303]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:03:19.234786 25228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7646288466ab42d997f16c8b73303e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:19.234787 25385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7646288466ab42d997f16c8b73303e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:19.234802 25542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7646288466ab42d997f16c8b73303e9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:19.237637 25635 tablet_bootstrap.cc:492] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: Bootstrap starting.
I20260810 23:03:19.237671 25634 tablet_bootstrap.cc:492] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: Bootstrap starting.
I20260810 23:03:19.238041 25635 tablet_bootstrap.cc:654] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:19.238175 25634 tablet_bootstrap.cc:654] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:19.238566 25636 tablet_bootstrap.cc:492] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: Bootstrap starting.
I20260810 23:03:19.238902 25634 tablet_bootstrap.cc:492] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: No bootstrap required, opened a new log
I20260810 23:03:19.238960 25634 ts_tablet_manager.cc:1397] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:19.239092 25636 tablet_bootstrap.cc:654] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:19.239161 25634 raft_consensus.cc:348] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.239259 25634 raft_consensus.cc:374] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:19.239280 25634 raft_consensus.cc:729] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cf7f60daae34291af3a2f217ca4d4c2, State: Initialized, Role: FOLLOWER
I20260810 23:03:19.239336 25634 consensus_queue.cc:260] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.239450 25634 ts_tablet_manager.cc:1428] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:19.239602 25640 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.239719 25640 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:19.239820 25636 tablet_bootstrap.cc:492] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: No bootstrap required, opened a new log
I20260810 23:03:19.239876 25636 ts_tablet_manager.cc:1397] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:19.240026 25636 raft_consensus.cc:348] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.240096 25636 raft_consensus.cc:374] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:19.240141 25636 raft_consensus.cc:729] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce35ee3b9f29449ba869e5477c99b753, State: Initialized, Role: FOLLOWER
I20260810 23:03:19.240182 25636 consensus_queue.cc:260] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.240279 25636 ts_tablet_manager.cc:1428] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:19.240324 25642 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.240408 25642 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:19.240639 25635 tablet_bootstrap.cc:492] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: No bootstrap required, opened a new log
I20260810 23:03:19.240695 25635 ts_tablet_manager.cc:1397] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260810 23:03:19.240792 25635 raft_consensus.cc:348] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.240862 25635 raft_consensus.cc:374] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:19.240885 25635 raft_consensus.cc:729] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8efa24a8ab33453fab13611bb6281948, State: Initialized, Role: FOLLOWER
I20260810 23:03:19.240929 25635 consensus_queue.cc:260] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.241032 25635 ts_tablet_manager.cc:1428] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:19.241073 25641 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.241166 25641 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:19.302577 25642 raft_consensus.cc:482] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:19.302664 25642 raft_consensus.cc:504] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.302812 25642 leader_election.cc:288] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cf7f60daae34291af3a2f217ca4d4c2 (127.23.56.66:32991), 8efa24a8ab33453fab13611bb6281948 (127.23.56.67:42633)
I20260810 23:03:19.303056 25408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7646288466ab42d997f16c8b73303e9d" candidate_uuid: "ce35ee3b9f29449ba869e5477c99b753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" is_pre_election: true
I20260810 23:03:19.303061 25566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7646288466ab42d997f16c8b73303e9d" candidate_uuid: "ce35ee3b9f29449ba869e5477c99b753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efa24a8ab33453fab13611bb6281948" is_pre_election: true
I20260810 23:03:19.303187 25408 raft_consensus.cc:2454] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce35ee3b9f29449ba869e5477c99b753 in term 0.
I20260810 23:03:19.303207 25566 raft_consensus.cc:2454] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce35ee3b9f29449ba869e5477c99b753 in term 0.
I20260810 23:03:19.303364 25170 leader_election.cc:302] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8efa24a8ab33453fab13611bb6281948, ce35ee3b9f29449ba869e5477c99b753; no voters:
I20260810 23:03:19.303483 25642 raft_consensus.cc:2781] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:19.303539 25642 raft_consensus.cc:482] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:19.303566 25642 raft_consensus.cc:3037] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:19.304234 25642 raft_consensus.cc:504] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.304383 25642 leader_election.cc:288] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [CANDIDATE]: Term 1 election: Requested vote from peers 7cf7f60daae34291af3a2f217ca4d4c2 (127.23.56.66:32991), 8efa24a8ab33453fab13611bb6281948 (127.23.56.67:42633)
I20260810 23:03:19.304589 25408 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7646288466ab42d997f16c8b73303e9d" candidate_uuid: "ce35ee3b9f29449ba869e5477c99b753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cf7f60daae34291af3a2f217ca4d4c2"
I20260810 23:03:19.304610 25566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7646288466ab42d997f16c8b73303e9d" candidate_uuid: "ce35ee3b9f29449ba869e5477c99b753" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8efa24a8ab33453fab13611bb6281948"
I20260810 23:03:19.304677 25566 raft_consensus.cc:3037] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:19.304677 25408 raft_consensus.cc:3037] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:19.305469 25566 raft_consensus.cc:2454] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce35ee3b9f29449ba869e5477c99b753 in term 1.
I20260810 23:03:19.305471 25408 raft_consensus.cc:2454] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce35ee3b9f29449ba869e5477c99b753 in term 1.
I20260810 23:03:19.305636 25170 leader_election.cc:302] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8efa24a8ab33453fab13611bb6281948, ce35ee3b9f29449ba869e5477c99b753; no voters:
I20260810 23:03:19.305729 25642 raft_consensus.cc:2781] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:19.305783 25642 raft_consensus.cc:686] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [term 1 LEADER]: Becoming Leader. State: Replica: ce35ee3b9f29449ba869e5477c99b753, State: Running, Role: LEADER
I20260810 23:03:19.305846 25642 consensus_queue.cc:237] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } }
I20260810 23:03:19.306164 25668 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: TxnStatusTablet state changed. Reason: New leader ce35ee3b9f29449ba869e5477c99b753. Latest consensus state: current_term: 1 leader_uuid: "ce35ee3b9f29449ba869e5477c99b753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.306249 25668 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:03:19.306427 25683 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:03:19.306551 25683 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:03:19.306536 25069 catalog_manager.cc:5495] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce35ee3b9f29449ba869e5477c99b753 (127.23.56.65). New cstate: current_term: 1 leader_uuid: "ce35ee3b9f29449ba869e5477c99b753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } health_report { overall_health: UNKNOWN } } }
I20260810 23:03:19.316270 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.309359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52936 (local address 127.23.56.65:45731)
0810 23:03:19.309406 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:19.309409 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:19.309418 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:19.312760 (+ 3342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:19.312761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:19.312783 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:19.312872 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:19.313085 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.313753 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.314989 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.315510 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.315619 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.315830 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:19.315942 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:19.316017 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:19.316131 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 23:03:19.316155 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:03:19.316265 25676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.309288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:45731 (local address 127.0.0.1:52936)
0810 23:03:19.309361 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:19.309373 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:19.309422 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:19.312937 (+ 3515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:19.312939 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:19.312940 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:19.313000 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.313002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.313863 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.313865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.314829 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.314833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.315622 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.315623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.315689 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.315708 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:19.316051 (+ 343us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:19.316064 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:03:19.316114 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 23:03:19.316170 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 23:03:19.317431 25408 raft_consensus.cc:1261] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Refusing update from remote peer ce35ee3b9f29449ba869e5477c99b753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:19.317436 25566 raft_consensus.cc:1261] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Refusing update from remote peer ce35ee3b9f29449ba869e5477c99b753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:19.317690 25642 consensus_queue.cc:1035] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:19.317842 25668 consensus_queue.cc:1035] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:19.318506 25640 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: TxnStatusTablet state changed. Reason: New leader ce35ee3b9f29449ba869e5477c99b753. Latest consensus state: current_term: 1 leader_uuid: "ce35ee3b9f29449ba869e5477c99b753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.318765 25640 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:19.318965 25641 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: TxnStatusTablet state changed. Reason: New leader ce35ee3b9f29449ba869e5477c99b753. Latest consensus state: current_term: 1 leader_uuid: "ce35ee3b9f29449ba869e5477c99b753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.319056 25641 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:19.319556 25642 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce35ee3b9f29449ba869e5477c99b753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.319629 25642 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:03:19.320008 25668 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ce35ee3b9f29449ba869e5477c99b753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.320096 25668 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P ce35ee3b9f29449ba869e5477c99b753: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:03:19.320508 25650 mvcc.cc:205] Tried to move back new op lower bound from 7317106685202255872 to 7317106685157146624. Current Snapshot: MvccSnapshot[applied={T|T < 7317106685202255872}]
I20260810 23:03:19.320618 25641 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce35ee3b9f29449ba869e5477c99b753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.320684 25641 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P 8efa24a8ab33453fab13611bb6281948: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:19.323689 25653 mvcc.cc:205] Tried to move back new op lower bound from 7317106685202255872 to 7317106685157146624. Current Snapshot: MvccSnapshot[applied={T|T < 7317106685202255872 or (T in {7317106685202255872})}]
I20260810 23:03:19.323875 25640 tablet_replica.cc:439] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ce35ee3b9f29449ba869e5477c99b753" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce35ee3b9f29449ba869e5477c99b753" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 45731 } } peers { permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 } } peers { permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 } } }
I20260810 23:03:19.323963 25640 tablet_replica.cc:442] T 7646288466ab42d997f16c8b73303e9d P 7cf7f60daae34291af3a2f217ca4d4c2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:19.326922 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.324684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.23.56.65:45731)
0810 23:03:19.324720 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:19.324721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:19.324730 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:19.324773 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:19.324774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:19.324781 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:19.324824 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:19.324927 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.325364 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.326377 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.326544 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.326592 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.326738 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:19.326772 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:19.326810 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:19.326862 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 23:03:19.326877 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:03:19.332190 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.329468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:45731 (local address 127.23.56.65:40247)
0810 23:03:19.329623 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:19.329633 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:19.329694 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:19.329818 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:19.329819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:19.329820 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:19.329874 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.329875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.330725 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.330726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.331376 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:19.331378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.331818 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.331818 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:19.331862 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.331874 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:19.332046 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:19.332048 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:03:19.332078 (+ 30us) client_negotiation.cc:240] Negotiation successful
0810 23:03:19.332108 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260810 23:03:19.332192 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.329526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:40247 (local address 127.23.56.65:45731)
0810 23:03:19.329564 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:19.329565 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:19.329704 (+ 139us) server_negotiation.cc:407] Connection header received
0810 23:03:19.329728 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:19.329729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:19.329739 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:19.329800 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:19.329940 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.330641 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.331446 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.331746 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.331818 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.331956 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:19.331989 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:19.332041 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:19.332093 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 23:03:19.332110 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:03:19.333952 25566 raft_consensus.cc:1261] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Refusing update from remote peer ce35ee3b9f29449ba869e5477c99b753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:19.333974 25408 raft_consensus.cc:1261] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Refusing update from remote peer ce35ee3b9f29449ba869e5477c99b753: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:19.334161 25668 consensus_queue.cc:1035] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cf7f60daae34291af3a2f217ca4d4c2" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 32991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:19.334263 25692 consensus_queue.cc:1035] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8efa24a8ab33453fab13611bb6281948" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:19.346405 25068 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:19.346510 25068 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:19.347209 25068 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 514f9d039f564efd806bca3fabcdbdf3: Sending DeleteTablet for 3 replicas of tablet 90e9be3d51da439d9278ec2208be7568
I20260810 23:03:19.347523 25385 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e9be3d51da439d9278ec2208be7568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:19 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:46526
I20260810 23:03:19.347582 25542 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e9be3d51da439d9278ec2208be7568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:19 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:57084
I20260810 23:03:19.347617 25660 tablet_replica.cc:330] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: stopping tablet replica
I20260810 23:03:19.347661 25660 raft_consensus.cc:2229] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:19.347673 25662 tablet_replica.cc:330] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: stopping tablet replica
I20260810 23:03:19.347690 25660 raft_consensus.cc:2258] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:19.347723 25662 raft_consensus.cc:2229] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:19.347757 25662 raft_consensus.cc:2258] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:19.347952 25660 ts_tablet_manager.cc:1905] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:19.348106 25227 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e9be3d51da439d9278ec2208be7568 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:19 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:52934
I20260810 23:03:19.348235 25661 tablet_replica.cc:330] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: stopping tablet replica
I20260810 23:03:19.348306 25661 raft_consensus.cc:2229] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:19.348402 25661 raft_consensus.cc:2258] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:19.348706 25662 ts_tablet_manager.cc:1905] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:19.348796 25661 ts_tablet_manager.cc:1905] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:19.350237 25662 ts_tablet_manager.cc:1918] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:19.350286 25662 log.cc:1196] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/90e9be3d51da439d9278ec2208be7568
I20260810 23:03:19.350383 25661 ts_tablet_manager.cc:1918] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:19.350427 25661 log.cc:1196] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/90e9be3d51da439d9278ec2208be7568
I20260810 23:03:19.350492 25662 ts_tablet_manager.cc:1939] T 90e9be3d51da439d9278ec2208be7568 P 8efa24a8ab33453fab13611bb6281948: Deleting consensus metadata
I20260810 23:03:19.350630 25661 ts_tablet_manager.cc:1939] T 90e9be3d51da439d9278ec2208be7568 P ce35ee3b9f29449ba869e5477c99b753: Deleting consensus metadata
I20260810 23:03:19.350728 25660 ts_tablet_manager.cc:1918] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:19.350801 25660 log.cc:1196] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/90e9be3d51da439d9278ec2208be7568
I20260810 23:03:19.351012 25660 ts_tablet_manager.cc:1939] T 90e9be3d51da439d9278ec2208be7568 P 7cf7f60daae34291af3a2f217ca4d4c2: Deleting consensus metadata
I20260810 23:03:19.351404 25052 catalog_manager.cc:4841] TS 8efa24a8ab33453fab13611bb6281948 (127.23.56.67:42633): tablet 90e9be3d51da439d9278ec2208be7568 (table test-table [id=b73c8d1f6c0a403f86ca9208248e1c73]) successfully deleted
I20260810 23:03:19.351480 25052 catalog_manager.cc:4841] TS 7cf7f60daae34291af3a2f217ca4d4c2 (127.23.56.66:32991): tablet 90e9be3d51da439d9278ec2208be7568 (table test-table [id=b73c8d1f6c0a403f86ca9208248e1c73]) successfully deleted
I20260810 23:03:19.351538 25052 catalog_manager.cc:4841] TS ce35ee3b9f29449ba869e5477c99b753 (127.23.56.65:45731): tablet 90e9be3d51da439d9278ec2208be7568 (table test-table [id=b73c8d1f6c0a403f86ca9208248e1c73]) successfully deleted
Aug 10 23:03:19 dist-test-slave-1xrh krb5kdc[25018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786402998, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:19.358554 25645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.348411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.23.56.65:45731)
0810 23:03:19.348453 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:19.348455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:19.348761 (+ 306us) server_negotiation.cc:407] Connection header received
0810 23:03:19.348886 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:19.348887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:19.348900 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:19.348934 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:19.349755 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.350548 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.351807 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.352112 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.352177 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.355422 (+ 3245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:19.355430 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:19.355430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:19.355438 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:19.357317 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:19.357666 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:19.357669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:19.357670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:19.357733 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:19.357957 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:19.357958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:19.357959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:19.358129 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:19.358300 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:19.358361 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 23:03:19.358442 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 23:03:19.359000 25298 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48002
I20260810 23:03:19.366675 25677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:19.360100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.23.56.126:45329)
0810 23:03:19.360152 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:19.360155 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:19.360179 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:03:19.360246 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:19.360247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:19.360257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:19.360324 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:19.360860 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.361386 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.362151 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:19.362427 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:19.362522 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:19.363845 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:19.363846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:19.363847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:19.363855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:19.365403 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:19.365805 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:19.365807 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:19.365807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:19.365885 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:19.366114 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:19.366115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:19.366116 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:19.366251 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:19.366374 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:19.366480 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 23:03:19.366566 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260810 23:03:19.366917 25068 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52548
I20260810 23:03:19.367583 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 25154
I20260810 23:03:19.376912 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 25314
I20260810 23:03:19.383827 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 25472
I20260810 23:03:19.391062 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 25034
2026-08-10T23:03:19Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2674 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:19 dist-test-slave-1xrh krb5kdc[25705](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:19 dist-test-slave-1xrh krb5kdc[25705](info): set up 2 sockets
Aug 10 23:03:19 dist-test-slave-1xrh krb5kdc[25705](info): commencing operation
krb5kdc: starting...
W20260810 23:03:20.488867 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:20 dist-test-slave-1xrh krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:20.610993 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38539
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:38539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:20.692137 25719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:20.692363 25719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:20.692400 25719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:20.693571 25719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:20.693634 25719 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:03:20.693655 25719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:20.693677 25719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:20.693706 25719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:20.693728 25719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:20.695273 25719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:38539
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:20.695542 25719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:20.695773 25719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:20.700419 25724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:20.700419 25728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:20.700549 25725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:20.700879 25719 server_base.cc:1034] running on GCE node
I20260810 23:03:20.701138 25719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:20.701403 25719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:20.702559 25719 hybrid_clock.cc:648] HybridClock initialized: now 1786403000702550 us; error 33 us; skew 500 ppm
Aug 10 23:03:20 dist-test-slave-1xrh krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403000, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:20.704973 25719 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:20.705462 25719 webserver.cc:460] Webserver started at http://127.23.56.126:45813/ using document root <none> and password file <none>
I20260810 23:03:20.705703 25719 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:20.705767 25719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:20.705911 25719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:20.707082 25719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "7a21c6e3b79e4d95b0d7964011168dfd"
format_stamp: "Formatted at 2026-08-10 23:03:20 on dist-test-slave-1xrh"
server_key: "bb7c53f144202fe28267d26a53ee2190"
server_key_iv: "8240ade5439b454c810040d3c7c28af6"
server_key_version: "encryptionkey@0"
I20260810 23:03:20.707518 25719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "7a21c6e3b79e4d95b0d7964011168dfd"
format_stamp: "Formatted at 2026-08-10 23:03:20 on dist-test-slave-1xrh"
server_key: "bb7c53f144202fe28267d26a53ee2190"
server_key_iv: "8240ade5439b454c810040d3c7c28af6"
server_key_version: "encryptionkey@0"
I20260810 23:03:20.708931 25719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:20.709609 25735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:20.709774 25719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:20.709870 25719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "7a21c6e3b79e4d95b0d7964011168dfd"
format_stamp: "Formatted at 2026-08-10 23:03:20 on dist-test-slave-1xrh"
server_key: "bb7c53f144202fe28267d26a53ee2190"
server_key_iv: "8240ade5439b454c810040d3c7c28af6"
server_key_version: "encryptionkey@0"
I20260810 23:03:20.709946 25719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:20.734212 25719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:20.736344 25719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:20.736538 25719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:20.741914 25719 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:38539
I20260810 23:03:20.741964 25809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:38539 every 8 connection(s)
I20260810 23:03:20.742363 25719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:20.743036 25810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:20.745338 25810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: Bootstrap starting.
I20260810 23:03:20.745905 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 25719
I20260810 23:03:20.746016 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:20.746002 25810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:20.746204 23777 external_mini_cluster.cc:1428] Setting key 915679db6e0a05c8a84df84079c40bba
I20260810 23:03:20.746356 25810 log.cc:824] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:20.747251 25810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: No bootstrap required, opened a new log
I20260810 23:03:20.749747 25810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a21c6e3b79e4d95b0d7964011168dfd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38539 } }
I20260810 23:03:20.749909 25810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:20.749940 25810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a21c6e3b79e4d95b0d7964011168dfd, State: Initialized, Role: FOLLOWER
I20260810 23:03:20.750075 25810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a21c6e3b79e4d95b0d7964011168dfd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38539 } }
I20260810 23:03:20.750164 25810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:20.750187 25810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:20.750268 25810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:20.750826 25810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a21c6e3b79e4d95b0d7964011168dfd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38539 } }
I20260810 23:03:20.750916 25810 leader_election.cc:302] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a21c6e3b79e4d95b0d7964011168dfd; no voters:
I20260810 23:03:20.751142 25810 leader_election.cc:288] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:20.751192 25816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:20.751411 25810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:20.751401 25816 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [term 1 LEADER]: Becoming Leader. State: Replica: 7a21c6e3b79e4d95b0d7964011168dfd, State: Running, Role: LEADER
I20260810 23:03:20.752131 25816 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a21c6e3b79e4d95b0d7964011168dfd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38539 } }
I20260810 23:03:20.752786 25818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a21c6e3b79e4d95b0d7964011168dfd. Latest consensus state: current_term: 1 leader_uuid: "7a21c6e3b79e4d95b0d7964011168dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a21c6e3b79e4d95b0d7964011168dfd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38539 } } }
I20260810 23:03:20.752874 25818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [sys.catalog]: This master's current role is: LEADER
Aug 10 23:03:20 dist-test-slave-1xrh krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:20.753006 25817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a21c6e3b79e4d95b0d7964011168dfd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a21c6e3b79e4d95b0d7964011168dfd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38539 } } }
W20260810 23:03:20.753413 25833 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 23:03:20.754280 25817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd [sys.catalog]: This master's current role is: LEADER
W20260810 23:03:20.754290 25833 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:03:20.754391 25834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:20.754588 25834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:20.756095 25834 catalog_manager.cc:1347] Generated new cluster ID: 524c5351295f44ed911bd5b8c3fef912
I20260810 23:03:20.756145 25834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:20.758373 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:20.746773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.23.56.126:38539)
0810 23:03:20.746965 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:20.746975 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:20.747008 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:03:20.747669 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:20.747679 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:20.747741 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:20.747898 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:20.748542 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.749991 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.751115 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.751734 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.751843 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:20.754675 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:20.754680 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:20.754680 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:20.754692 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:20.756620 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:20.757091 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:20.757095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:20.757096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:20.757214 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:20.757437 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:20.757439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:20.757439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:20.757773 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:20.757938 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:20.758069 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 23:03:20.758202 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":81,"threads_started":1}
I20260810 23:03:20.784067 25834 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:20.784859 25834 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:20.788882 25834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: Generated new TSK 0
I20260810 23:03:20.789079 25834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:20.821696 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38539
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:20.903038 25842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:20.903244 25842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:20.903267 25842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:20.904349 25842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:20.904383 25842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:20.904404 25842 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:03:20.904417 25842 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:03:20.904433 25842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:20.905812 25842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.56.126:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:20.906028 25842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:20.906193 25842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:20.910863 25851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:20.910883 25849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:20.910897 25848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:20.911049 25842 server_base.cc:1034] running on GCE node
I20260810 23:03:20.911382 25842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:20.911607 25842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:20.912757 25842 hybrid_clock.cc:648] HybridClock initialized: now 1786403000912741 us; error 44 us; skew 500 ppm
Aug 10 23:03:20 dist-test-slave-1xrh krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403000, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:20.915088 25842 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:20.915526 25842 webserver.cc:460] Webserver started at http://127.23.56.65:42545/ using document root <none> and password file <none>
I20260810 23:03:20.915735 25842 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:20.915796 25842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:20.915937 25842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:20.917017 25842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "06631da6424b44c2925f39a85ee8b4ee"
format_stamp: "Formatted at 2026-08-10 23:03:20 on dist-test-slave-1xrh"
server_key: "88e2dce7c27c54c1080112cf93591d2f"
server_key_iv: "5a80c7422303e94d3f9db08440e85f81"
server_key_version: "encryptionkey@0"
I20260810 23:03:20.917449 25842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "06631da6424b44c2925f39a85ee8b4ee"
format_stamp: "Formatted at 2026-08-10 23:03:20 on dist-test-slave-1xrh"
server_key: "88e2dce7c27c54c1080112cf93591d2f"
server_key_iv: "5a80c7422303e94d3f9db08440e85f81"
server_key_version: "encryptionkey@0"
I20260810 23:03:20.918853 25842 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:20.919585 25860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:20.919741 25842 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:20.919840 25842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "06631da6424b44c2925f39a85ee8b4ee"
format_stamp: "Formatted at 2026-08-10 23:03:20 on dist-test-slave-1xrh"
server_key: "88e2dce7c27c54c1080112cf93591d2f"
server_key_iv: "5a80c7422303e94d3f9db08440e85f81"
server_key_version: "encryptionkey@0"
I20260810 23:03:20.919915 25842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:20.939684 25842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:20.941501 25842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:20.941658 25842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:20.942150 25842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:20.942209 25842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:20.942251 25842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:20.942287 25842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:03:20 dist-test-slave-1xrh krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403000, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:20.949159 25842 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:36289
I20260810 23:03:20.949287 25985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:36289 every 8 connection(s)
I20260810 23:03:20.949554 25842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:20.952147 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:20.942642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:51563 (local address 127.23.56.126:38539)
0810 23:03:20.942712 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:20.942715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:20.942954 (+ 239us) server_negotiation.cc:407] Connection header received
0810 23:03:20.943469 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:20.943471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:20.943483 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:20.943542 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:20.944334 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.945186 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.946755 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.947099 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.947204 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:20.949492 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:20.949494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:20.949494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:20.949501 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:20.950945 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:20.951367 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:20.951368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:20.951369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:20.951420 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:20.951610 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:20.951611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:20.951611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:20.951699 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:20.951743 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:20.952012 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 23:03:20.952084 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:03:20.952263 25869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:20.942598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38539 (local address 127.23.56.65:51563)
0810 23:03:20.942849 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:20.942877 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:20.943368 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:20.943609 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:20.943613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:20.943942 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:20.944220 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:20.944227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.945292 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.945293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:20.946673 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:20.946677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.947208 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.947209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:20.947367 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:20.947925 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:20.947933 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:20.949397 (+ 1464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:20.951022 (+ 1625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:20.951027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:20.951028 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:20.951315 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:20.951474 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:20.951475 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:20.951475 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:20.951572 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:20.951744 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:20.951748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:20.951845 (+ 97us) client_negotiation.cc:769] Sending connection context
0810 23:03:20.951973 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 23:03:20.952095 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":88,"threads_started":1}
I20260810 23:03:20.952739 25986 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38539
I20260810 23:03:20.952850 25986 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:20.953069 25986 heartbeater.cc:511] Master 127.23.56.126:38539 requested a full tablet report, sending...
I20260810 23:03:20.953550 25757 ts_manager.cc:194] Registered new tserver with Master: 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289)
I20260810 23:03:20.954375 25757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:51563
I20260810 23:03:20.956712 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 25842
I20260810 23:03:20.956822 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:20.957050 23777 external_mini_cluster.cc:1428] Setting key a2c8f6cde8567eeb222b38e5b9733705
I20260810 23:03:20.959676 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:20.953822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:40055 (local address 127.23.56.126:38539)
0810 23:03:20.953859 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:20.953860 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:20.953930 (+ 70us) server_negotiation.cc:407] Connection header received
0810 23:03:20.953991 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:20.953992 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:20.954002 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:20.954068 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:20.954312 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.955001 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.955855 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.956022 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.956071 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:20.956958 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:20.956959 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:20.956960 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:20.956985 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:20.958618 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:20.958913 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:20.958914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:20.958914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:20.958973 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:20.959104 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:20.959105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:20.959106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:20.959253 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:20.959326 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:20.959451 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 23:03:20.959551 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:03:20.959693 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:20.953763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38539 (local address 127.23.56.65:40055)
0810 23:03:20.953875 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:20.953885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:20.953937 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:20.954072 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:20.954072 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:20.954212 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:20.954255 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:20.954256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.955105 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.955106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:20.955785 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:20.955788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:20.956079 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:20.956079 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:20.956131 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:20.956547 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:20.956552 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:20.956858 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:20.958694 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:20.958695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:20.958695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:20.958847 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:20.959029 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:20.959030 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:20.959030 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:20.959068 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:20.959322 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:20.959323 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:20.959357 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 23:03:20.959409 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 23:03:20.959479 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:20.983675 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38539
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:21.065181 25996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:21.065373 25996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:21.065397 25996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:21.066522 25996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:21.066565 25996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:21.066586 25996 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:03:21.066630 25996 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:03:21.066663 25996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:21.068156 25996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.56.126:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:21.068401 25996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:21.068630 25996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:21.073635 26006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:21.073698 25996 server_base.cc:1034] running on GCE node
W20260810 23:03:21.073609 26004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:21.073596 26003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:21.074070 25996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:21.074323 25996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:21.075485 25996 hybrid_clock.cc:648] HybridClock initialized: now 1786403001075467 us; error 30 us; skew 500 ppm
Aug 10 23:03:21 dist-test-slave-1xrh krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:21.077857 25996 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:21.078296 25996 webserver.cc:460] Webserver started at http://127.23.56.66:34111/ using document root <none> and password file <none>
I20260810 23:03:21.078521 25996 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:21.078585 25996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:21.078727 25996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:21.079830 25996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "38b8feaefef64d238daad3f374cbb9ca"
format_stamp: "Formatted at 2026-08-10 23:03:21 on dist-test-slave-1xrh"
server_key: "f10e4abdce15f1c364307dba80fcebf0"
server_key_iv: "9013f2e52c8787fcb7e3c83a8f1cd4d2"
server_key_version: "encryptionkey@0"
I20260810 23:03:21.080184 25996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "38b8feaefef64d238daad3f374cbb9ca"
format_stamp: "Formatted at 2026-08-10 23:03:21 on dist-test-slave-1xrh"
server_key: "f10e4abdce15f1c364307dba80fcebf0"
server_key_iv: "9013f2e52c8787fcb7e3c83a8f1cd4d2"
server_key_version: "encryptionkey@0"
I20260810 23:03:21.081596 25996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:21.082296 26013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:21.082450 25996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:21.082547 25996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "38b8feaefef64d238daad3f374cbb9ca"
format_stamp: "Formatted at 2026-08-10 23:03:21 on dist-test-slave-1xrh"
server_key: "f10e4abdce15f1c364307dba80fcebf0"
server_key_iv: "9013f2e52c8787fcb7e3c83a8f1cd4d2"
server_key_version: "encryptionkey@0"
I20260810 23:03:21.082614 25996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:21.091755 25996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:21.093545 25996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:21.093693 25996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:21.094182 25996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:21.094244 25996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:21.094295 25996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:21.094348 25996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:21.101094 25996 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:36231
I20260810 23:03:21.101167 26139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:36231 every 8 connection(s)
Aug 10 23:03:21 dist-test-slave-1xrh krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:21.101478 25996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:21.104238 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.094654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:52135 (local address 127.23.56.126:38539)
0810 23:03:21.094713 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.094715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.094919 (+ 204us) server_negotiation.cc:407] Connection header received
0810 23:03:21.095443 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.095445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.095460 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.095546 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:21.096299 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.097154 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.098758 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.099140 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.099221 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.101920 (+ 2699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:21.101922 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:21.101922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:21.101931 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:21.103100 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.103537 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.103537 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.103538 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.103588 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.103752 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.103752 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.103753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.103860 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:21.103910 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.104113 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.104175 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 23:03:21.104458 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.094633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38539 (local address 127.23.56.66:52135)
0810 23:03:21.094791 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.094830 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.095346 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.095593 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.095596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.095904 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:21.096191 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.096200 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.097276 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.097277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.098648 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.098652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.099228 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.099229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.099378 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.099913 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:21.099925 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:21.101832 (+ 1907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:21.103255 (+ 1423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.103261 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.103263 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.103468 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.103627 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.103627 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.103627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.103706 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.103912 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:21.103917 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:21.104024 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.104103 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.104214 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260810 23:03:21.104975 26140 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38539
I20260810 23:03:21.105055 26140 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:21.105242 26140 heartbeater.cc:511] Master 127.23.56.126:38539 requested a full tablet report, sending...
I20260810 23:03:21.105646 25757 ts_manager.cc:194] Registered new tserver with Master: 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231)
I20260810 23:03:21.106153 25757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:52135
I20260810 23:03:21.108539 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 25996
I20260810 23:03:21.108641 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:21.108817 23777 external_mini_cluster.cc:1428] Setting key db246097e43fdbe94e1a5790aad6c1da
I20260810 23:03:21.111510 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.106084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:46745 (local address 127.23.56.126:38539)
0810 23:03:21.106125 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.106127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.106223 (+ 96us) server_negotiation.cc:407] Connection header received
0810 23:03:21.106291 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.106292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.106301 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.106362 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:21.106706 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.107219 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.107839 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.108022 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.108069 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.108987 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:21.108989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:21.108989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:21.108996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:21.110363 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.110701 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.110702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.110702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.110759 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.110944 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.110945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.110946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.111084 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:21.111189 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.111341 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.111418 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:03:21.111521 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.106007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38539 (local address 127.23.56.66:46745)
0810 23:03:21.106147 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.106158 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.106222 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.106364 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.106365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.106570 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:21.106633 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.106635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.107301 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.107302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.107765 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.107766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.108080 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.108081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.108135 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.108556 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:21.108560 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:21.108916 (+ 356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:21.110462 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.110463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.110463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.110632 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.110822 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.110822 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.110823 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.110879 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.111183 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:21.111184 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:21.111232 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.111313 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.111415 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:21.134047 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38539
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:21.215356 26154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:21.215554 26154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:21.215579 26154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:21.216630 26154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:21.216665 26154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:21.216684 26154 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 23:03:21.216696 26154 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:03:21.216725 26154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:21.218116 26154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.56.126:38539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26154
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:21.218299 26154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:21.218462 26154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:21.223276 26162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:21.223287 26164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:21.223296 26160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:21.223523 26154 server_base.cc:1034] running on GCE node
I20260810 23:03:21.223737 26154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:21.223979 26154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:21.225163 26154 hybrid_clock.cc:648] HybridClock initialized: now 1786403001225145 us; error 29 us; skew 500 ppm
Aug 10 23:03:21 dist-test-slave-1xrh krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:21.227937 26154 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:21.228417 26154 webserver.cc:460] Webserver started at http://127.23.56.67:46127/ using document root <none> and password file <none>
I20260810 23:03:21.228644 26154 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:21.228709 26154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:21.228859 26154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:21.229952 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "0454b06429984d8585357a1f1a5c7d95"
format_stamp: "Formatted at 2026-08-10 23:03:21 on dist-test-slave-1xrh"
server_key: "75ad8e850747c8c591734c9d8317dded"
server_key_iv: "6449c37aff1e22b33645a5d450efbc62"
server_key_version: "encryptionkey@0"
I20260810 23:03:21.230361 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "0454b06429984d8585357a1f1a5c7d95"
format_stamp: "Formatted at 2026-08-10 23:03:21 on dist-test-slave-1xrh"
server_key: "75ad8e850747c8c591734c9d8317dded"
server_key_iv: "6449c37aff1e22b33645a5d450efbc62"
server_key_version: "encryptionkey@0"
I20260810 23:03:21.231837 26154 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:21.232583 26171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:21.232751 26154 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:21.232853 26154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "0454b06429984d8585357a1f1a5c7d95"
format_stamp: "Formatted at 2026-08-10 23:03:21 on dist-test-slave-1xrh"
server_key: "75ad8e850747c8c591734c9d8317dded"
server_key_iv: "6449c37aff1e22b33645a5d450efbc62"
server_key_version: "encryptionkey@0"
I20260810 23:03:21.232933 26154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:21.251935 26154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:21.254220 26154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:21.254365 26154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:21.254887 26154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:21.254951 26154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:21.254997 26154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:21.255024 26154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:21.261373 26154 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:40515
I20260810 23:03:21.261448 26300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:40515 every 8 connection(s)
I20260810 23:03:21.261758 26154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:03:21 dist-test-slave-1xrh krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403001, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:21.264941 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.255367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:48333 (local address 127.23.56.126:38539)
0810 23:03:21.255424 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.255427 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.255632 (+ 205us) server_negotiation.cc:407] Connection header received
0810 23:03:21.256180 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.256182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.256195 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.256243 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:21.257129 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.257928 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.259538 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.259787 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.259870 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.262381 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:21.262382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:21.262383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:21.262389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:21.263760 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.264189 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.264190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.264191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.264239 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.264447 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.264447 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.264448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.264553 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:21.264604 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.264819 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.264880 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 23:03:21.265172 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.255360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38539 (local address 127.23.56.67:48333)
0810 23:03:21.255519 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.255553 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.256071 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.256328 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.256332 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.256713 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:21.257003 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.257010 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.258061 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.258062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.259426 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.259429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.259881 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.259882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.260044 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.260559 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:21.260585 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:21.262298 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:21.263899 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.263905 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.263907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.264123 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.264295 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.264296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.264296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.264394 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.264612 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:21.264616 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:21.264720 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.264801 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.264953 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260810 23:03:21.265650 26301 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38539
I20260810 23:03:21.265735 26301 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:21.265920 26301 heartbeater.cc:511] Master 127.23.56.126:38539 requested a full tablet report, sending...
I20260810 23:03:21.266315 25758 ts_manager.cc:194] Registered new tserver with Master: 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515)
I20260810 23:03:21.266783 25758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:48333
I20260810 23:03:21.268784 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 26154
I20260810 23:03:21.268878 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:21.269047 23777 external_mini_cluster.cc:1428] Setting key 5f87a4af2d6de2efbb5966b7a93df7c7
I20260810 23:03:21.269765 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:03:21.272195 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.266752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:56363 (local address 127.23.56.126:38539)
0810 23:03:21.266794 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.266796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.266908 (+ 112us) server_negotiation.cc:407] Connection header received
0810 23:03:21.266978 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.266979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.266990 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.267050 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:21.267399 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.267858 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.268525 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.268722 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.268801 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.269721 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:21.269722 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:21.269723 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:21.269729 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:21.271073 (+ 1344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.271447 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.271448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.271449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.271501 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.271700 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.271701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.271701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.271805 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:21.271880 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.272044 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.272112 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:03:21.272212 26307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.266690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38539 (local address 127.23.56.67:56363)
0810 23:03:21.266849 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.266860 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.266922 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.267056 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.267056 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.267262 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:21.267326 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.267328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.267946 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.267947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.268448 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.268449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.268804 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.268805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.268881 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.269327 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:21.269332 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:21.269619 (+ 287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:21.271205 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.271206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.271206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.271377 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.271575 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.271576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.271576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.271637 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.271872 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:21.271873 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:21.271918 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.272005 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.272109 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":81,"threads_started":1}
Aug 10 23:03:21 dist-test-slave-1xrh krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:03:21 dist-test-slave-1xrh krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:21.293145 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.286570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52886 (local address 127.23.56.126:38539)
0810 23:03:21.286626 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.286628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.286640 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:03:21.286703 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.286704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.286717 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.286765 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:21.287375 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.287852 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.288532 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.288763 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.288832 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.291031 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:21.291032 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:21.291033 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:21.291039 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:21.292237 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.292536 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.292537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.292537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.292584 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.292733 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.292734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.292734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.292825 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:21.292895 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.293026 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.293083 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 23:03:21.294760 25758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:21.296427 25758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:21.302685 26319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.298458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:36231 (local address 127.0.0.1:33000)
0810 23:03:21.298744 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.298753 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.298808 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.299306 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.299307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.299309 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:21.299358 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.299360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.300626 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.300627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.301532 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.301534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.302439 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.302440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.302496 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.302508 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.302564 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.302598 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":100,"threads_started":1}
I20260810 23:03:21.302726 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.298912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33000 (local address 127.23.56.66:36231)
0810 23:03:21.299103 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.299108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.299118 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:21.299187 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.299190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.299205 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.299430 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:21.299505 (+ 75us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.300541 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.301609 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.302361 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.302441 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.302507 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.302578 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.302623 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":99,"threads_started":1}
I20260810 23:03:21.303031 26322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.298618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:40515 (local address 127.0.0.1:42800)
0810 23:03:21.299190 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.299201 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.299257 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.299516 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.299517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.299519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:21.299559 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.299560 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.300385 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.300386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.301755 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.301758 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.302642 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.302643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.302707 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.302712 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.302912 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.302957 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":36,"thread_start_us":84,"threads_started":1}
I20260810 23:03:21.303493 26066 tablet_service.cc:1467] Processing CreateTablet for tablet a0835c9444554d039476e0c7d7259ee8 (DEFAULT_TABLE table=test-table [id=336dd39259e244ceafe46a909df88563]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:21.303884 26066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0835c9444554d039476e0c7d7259ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:21.304055 26320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.298408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:36289 (local address 127.0.0.1:38668)
0810 23:03:21.298694 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.298738 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.298808 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.299037 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.299041 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.299043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:21.299108 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.299109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.300500 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.300501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.301756 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.301758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.303803 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.303804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.303867 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.303870 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.303932 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.303973 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":48,"thread_start_us":60,"threads_started":1}
I20260810 23:03:21.304126 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.298544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38668 (local address 127.23.56.65:36289)
0810 23:03:21.298702 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.298708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.298809 (+ 101us) server_negotiation.cc:407] Connection header received
0810 23:03:21.298898 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.298901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.298917 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.299023 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:21.299291 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.300407 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.302728 (+ 2321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.303624 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.303699 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.303753 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.303967 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.304013 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260810 23:03:21.304361 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.298729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42800 (local address 127.23.56.67:40515)
0810 23:03:21.299150 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.299155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.299315 (+ 160us) server_negotiation.cc:407] Connection header received
0810 23:03:21.299357 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.299360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.299374 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.299446 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:21.299642 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.300298 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.301842 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.302563 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.302634 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.302688 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.304219 (+ 1531us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.304266 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":70,"threads_started":1}
I20260810 23:03:21.304754 25913 tablet_service.cc:1467] Processing CreateTablet for tablet a0835c9444554d039476e0c7d7259ee8 (DEFAULT_TABLE table=test-table [id=336dd39259e244ceafe46a909df88563]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:21.304965 26228 tablet_service.cc:1467] Processing CreateTablet for tablet a0835c9444554d039476e0c7d7259ee8 (DEFAULT_TABLE table=test-table [id=336dd39259e244ceafe46a909df88563]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:21.305132 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0835c9444554d039476e0c7d7259ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:21.305264 26228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0835c9444554d039476e0c7d7259ee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:21.307210 26325 tablet_bootstrap.cc:492] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: Bootstrap starting.
I20260810 23:03:21.307843 26325 tablet_bootstrap.cc:654] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:21.308102 26325 log.cc:824] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:21.308749 26328 tablet_bootstrap.cc:492] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: Bootstrap starting.
I20260810 23:03:21.308830 26325 tablet_bootstrap.cc:492] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: No bootstrap required, opened a new log
I20260810 23:03:21.308876 26327 tablet_bootstrap.cc:492] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: Bootstrap starting.
I20260810 23:03:21.308900 26325 ts_tablet_manager.cc:1397] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 23:03:21.309576 26328 tablet_bootstrap.cc:654] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:21.309810 26327 tablet_bootstrap.cc:654] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:21.310063 26328 log.cc:824] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:21.310149 26327 log.cc:824] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:21.310828 26327 tablet_bootstrap.cc:492] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: No bootstrap required, opened a new log
I20260810 23:03:21.310828 26328 tablet_bootstrap.cc:492] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: No bootstrap required, opened a new log
I20260810 23:03:21.310891 26327 ts_tablet_manager.cc:1397] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:21.310891 26328 ts_tablet_manager.cc:1397] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:21.310904 26325 raft_consensus.cc:348] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.311074 26325 raft_consensus.cc:374] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:21.311112 26325 raft_consensus.cc:729] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b8feaefef64d238daad3f374cbb9ca, State: Initialized, Role: FOLLOWER
I20260810 23:03:21.311307 26325 consensus_queue.cc:260] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.311594 26325 ts_tablet_manager.cc:1428] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 23:03:21.311710 26140 heartbeater.cc:503] Master 127.23.56.126:38539 was elected leader, sending a full tablet report...
I20260810 23:03:21.312853 26328 raft_consensus.cc:348] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.312853 26327 raft_consensus.cc:348] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.312996 26328 raft_consensus.cc:374] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:21.313010 26327 raft_consensus.cc:374] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:21.313035 26328 raft_consensus.cc:729] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06631da6424b44c2925f39a85ee8b4ee, State: Initialized, Role: FOLLOWER
I20260810 23:03:21.313045 26327 raft_consensus.cc:729] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0454b06429984d8585357a1f1a5c7d95, State: Initialized, Role: FOLLOWER
I20260810 23:03:21.313144 26328 consensus_queue.cc:260] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.313144 26327 consensus_queue.cc:260] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.313393 26328 ts_tablet_manager.cc:1428] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:03:21.313393 26327 ts_tablet_manager.cc:1428] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:03:21.313445 25986 heartbeater.cc:503] Master 127.23.56.126:38539 was elected leader, sending a full tablet report...
I20260810 23:03:21.313526 26301 heartbeater.cc:503] Master 127.23.56.126:38539 was elected leader, sending a full tablet report...
W20260810 23:03:21.352097 26144 tablet.cc:2369] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:21.450683 25987 tablet.cc:2369] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:21.512349 26302 tablet.cc:2369] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:03:21.678462 26334 raft_consensus.cc:482] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:21.678645 26334 raft_consensus.cc:504] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.679075 26334 leader_election.cc:288] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289), 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231)
I20260810 23:03:21.679340 26333 raft_consensus.cc:482] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:21.679605 26333 raft_consensus.cc:504] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.680387 26333 leader_election.cc:288] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515), 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231)
I20260810 23:03:21.682976 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.679492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:43505 (local address 127.23.56.65:36289)
0810 23:03:21.679536 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.679538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.679743 (+ 205us) server_negotiation.cc:407] Connection header received
0810 23:03:21.679858 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.679859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.679869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.679930 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:21.680093 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.680781 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.682131 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.682626 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.682670 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.682678 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.682879 (+ 201us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.682911 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:03:21.683233 26335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.679392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:36289 (local address 127.23.56.67:43505)
0810 23:03:21.679665 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.679678 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.679774 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.679946 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.679947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.679949 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:21.680007 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.680008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.680869 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.680869 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.682047 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.682050 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.682733 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.682734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.682806 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.682810 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.683051 (+ 241us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.683090 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":123,"thread_start_us":100,"threads_started":1}
I20260810 23:03:21.683578 26181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.679392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:36231 (local address 127.23.56.67:39289)
0810 23:03:21.679467 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.679479 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.679553 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.679732 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.679733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.679735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:21.679813 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.679815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.680774 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.680775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.682021 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.682024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.682740 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.682741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.683413 (+ 672us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.683417 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.683480 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.683515 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":57}
I20260810 23:03:21.683615 26324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.679456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:39289 (local address 127.23.56.66:36231)
0810 23:03:21.679502 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.679504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.679547 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:03:21.679620 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.679622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.679634 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.679714 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:21.679901 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.680682 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.682117 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.682638 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.682701 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.682708 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.683505 (+ 797us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.683550 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:03:21.683954 25935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0835c9444554d039476e0c7d7259ee8" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06631da6424b44c2925f39a85ee8b4ee" is_pre_election: true
I20260810 23:03:21.684068 25935 raft_consensus.cc:2454] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 0.
I20260810 23:03:21.684087 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.680763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:40143 (local address 127.23.56.67:40515)
0810 23:03:21.681013 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.681014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.681021 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:03:21.681038 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.681039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.681048 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.681098 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:21.681276 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.681941 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.683217 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.683747 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.683829 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.683836 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.683998 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.684033 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260810 23:03:21.684147 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0835c9444554d039476e0c7d7259ee8" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca" is_pre_election: true
I20260810 23:03:21.684269 26337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.680174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:40515 (local address 127.23.56.65:40143)
0810 23:03:21.680323 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.680335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.680384 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.681135 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.681136 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.681138 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:21.681195 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.681197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.682038 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.682039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.683098 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.683100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.683832 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.683833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.683903 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.683905 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.684151 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.684195 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":57,"threads_started":1}
I20260810 23:03:21.684285 26089 raft_consensus.cc:2454] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 0.
I20260810 23:03:21.684319 26173 leader_election.cc:302] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0454b06429984d8585357a1f1a5c7d95, 06631da6424b44c2925f39a85ee8b4ee; no voters:
I20260810 23:03:21.684540 26334 raft_consensus.cc:2781] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:21.684604 26334 raft_consensus.cc:482] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:21.684638 26334 raft_consensus.cc:3037] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:21.684648 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0835c9444554d039476e0c7d7259ee8" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0454b06429984d8585357a1f1a5c7d95" is_pre_election: true
I20260810 23:03:21.684808 26338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.680178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:36231 (local address 127.23.56.65:44967)
0810 23:03:21.680453 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.680461 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.680499 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.680698 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.680698 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.680700 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:21.680759 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.680760 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.681636 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.681637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.682777 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.682780 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.684563 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.684564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.684634 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.684638 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.684696 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.684735 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":11,"thread_start_us":56,"threads_started":1}
I20260810 23:03:21.685016 26339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.680315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:44967 (local address 127.23.56.66:36231)
0810 23:03:21.680579 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.680581 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.680587 (+ 6us) server_negotiation.cc:407] Connection header received
0810 23:03:21.680608 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.680609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.680622 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.680693 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:21.680827 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.681540 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.682870 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.684472 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.684556 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.684565 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.684795 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.684842 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":49,"threads_started":1}
I20260810 23:03:21.685226 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0835c9444554d039476e0c7d7259ee8" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca" is_pre_election: true
I20260810 23:03:21.685344 26089 raft_consensus.cc:2454] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06631da6424b44c2925f39a85ee8b4ee in term 0.
I20260810 23:03:21.685555 25863 leader_election.cc:302] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06631da6424b44c2925f39a85ee8b4ee, 38b8feaefef64d238daad3f374cbb9ca; no voters:
I20260810 23:03:21.685675 26334 raft_consensus.cc:504] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.685688 26333 raft_consensus.cc:2781] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:21.685753 26333 raft_consensus.cc:482] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:21.685807 26333 raft_consensus.cc:3037] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:21.685834 26334 leader_election.cc:288] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 election: Requested vote from peers 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289), 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231)
I20260810 23:03:21.685853 26244 raft_consensus.cc:2379] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 06631da6424b44c2925f39a85ee8b4ee in current term 1: Already voted for candidate 0454b06429984d8585357a1f1a5c7d95 in this term.
I20260810 23:03:21.685964 26089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0835c9444554d039476e0c7d7259ee8" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca"
I20260810 23:03:21.686033 26089 raft_consensus.cc:3037] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:21.686050 25935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0835c9444554d039476e0c7d7259ee8" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06631da6424b44c2925f39a85ee8b4ee"
I20260810 23:03:21.686621 26333 raft_consensus.cc:504] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.686740 25935 raft_consensus.cc:2379] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0454b06429984d8585357a1f1a5c7d95 in current term 1: Already voted for candidate 06631da6424b44c2925f39a85ee8b4ee in this term.
I20260810 23:03:21.686761 26333 leader_election.cc:288] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 election: Requested vote from peers 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515), 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231)
I20260810 23:03:21.686849 26089 raft_consensus.cc:2454] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 1.
I20260810 23:03:21.687000 26087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0835c9444554d039476e0c7d7259ee8" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca"
I20260810 23:03:21.687069 26174 leader_election.cc:302] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0454b06429984d8585357a1f1a5c7d95, 38b8feaefef64d238daad3f374cbb9ca; no voters: 06631da6424b44c2925f39a85ee8b4ee
I20260810 23:03:21.687093 26087 raft_consensus.cc:2379] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 06631da6424b44c2925f39a85ee8b4ee in current term 1: Already voted for candidate 0454b06429984d8585357a1f1a5c7d95 in this term.
I20260810 23:03:21.687178 26334 raft_consensus.cc:2781] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:21.687227 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0835c9444554d039476e0c7d7259ee8" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0454b06429984d8585357a1f1a5c7d95"
I20260810 23:03:21.687325 26334 raft_consensus.cc:686] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 1 LEADER]: Becoming Leader. State: Replica: 0454b06429984d8585357a1f1a5c7d95, State: Running, Role: LEADER
I20260810 23:03:21.687417 26334 consensus_queue.cc:237] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } }
I20260810 23:03:21.687951 25862 leader_election.cc:302] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 06631da6424b44c2925f39a85ee8b4ee; no voters: 0454b06429984d8585357a1f1a5c7d95, 38b8feaefef64d238daad3f374cbb9ca
I20260810 23:03:21.688068 26333 raft_consensus.cc:2726] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260810 23:03:21.688287 25757 catalog_manager.cc:5495] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67). New cstate: current_term: 1 leader_uuid: "0454b06429984d8585357a1f1a5c7d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:03:21 dist-test-slave-1xrh krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403000, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:21.766309 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.759864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38539 (local address 127.0.0.1:52894)
0810 23:03:21.760030 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:21.760041 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:21.760113 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:21.760245 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:21.760246 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:21.760546 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:21.760611 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.760613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.761257 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.761258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.761765 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:21.761766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.762075 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.762076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:21.762123 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.762600 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:21.762604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:21.763949 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:21.765358 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.765364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.765366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.765568 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.765741 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:21.765741 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:21.765742 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:21.765796 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:21.766042 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:21.766053 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:21.766095 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 23:03:21.766149 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 23:03:21.766236 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260810 23:03:21.766427 25815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:21.759937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52894 (local address 127.23.56.126:38539)
0810 23:03:21.760020 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:21.760022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:21.760105 (+ 83us) server_negotiation.cc:407] Connection header received
0810 23:03:21.760166 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:21.760167 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:21.760176 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:21.760230 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:21.760671 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.761193 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.761828 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:21.762015 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:21.762065 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:21.764025 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:21.764027 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:21.764027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:21.764034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:21.765216 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.765632 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.765633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.765633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.765686 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:21.765844 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:21.765845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:21.765845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:21.765966 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:21.766018 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:21.766181 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:03:21.766287 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":45}
I20260810 23:03:21.768285 25757 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:52894:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:03:21.768481 25757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:21.770000 26228 tablet_service.cc:1467] Processing CreateTablet for tablet e06918c3f4de424fa020d7d301075ee9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=79e725799a744cfa8cb9d96bd8d34a71]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:03:21.769994 25913 tablet_service.cc:1467] Processing CreateTablet for tablet e06918c3f4de424fa020d7d301075ee9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=79e725799a744cfa8cb9d96bd8d34a71]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:03:21.770056 26066 tablet_service.cc:1467] Processing CreateTablet for tablet e06918c3f4de424fa020d7d301075ee9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=79e725799a744cfa8cb9d96bd8d34a71]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:03:21.770182 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06918c3f4de424fa020d7d301075ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:21.770182 26228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06918c3f4de424fa020d7d301075ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:21.770241 26066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e06918c3f4de424fa020d7d301075ee9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:21.771557 26325 tablet_bootstrap.cc:492] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: Bootstrap starting.
I20260810 23:03:21.771723 26328 tablet_bootstrap.cc:492] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: Bootstrap starting.
I20260810 23:03:21.771976 26325 tablet_bootstrap.cc:654] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:21.772436 26328 tablet_bootstrap.cc:654] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:21.772642 26325 tablet_bootstrap.cc:492] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: No bootstrap required, opened a new log
I20260810 23:03:21.772701 26325 ts_tablet_manager.cc:1397] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:21.772773 26327 tablet_bootstrap.cc:492] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: Bootstrap starting.
I20260810 23:03:21.772876 26325 raft_consensus.cc:348] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:21.772974 26325 raft_consensus.cc:374] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:21.773007 26325 raft_consensus.cc:729] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b8feaefef64d238daad3f374cbb9ca, State: Initialized, Role: FOLLOWER
I20260810 23:03:21.773047 26325 consensus_queue.cc:260] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:21.773136 26325 ts_tablet_manager.cc:1428] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:21.773156 26327 tablet_bootstrap.cc:654] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:21.773406 26331 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:21.773524 26331 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:21.774076 26328 tablet_bootstrap.cc:492] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: No bootstrap required, opened a new log
I20260810 23:03:21.774135 26328 ts_tablet_manager.cc:1397] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 23:03:21.774154 26327 tablet_bootstrap.cc:492] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: No bootstrap required, opened a new log
I20260810 23:03:21.774204 26327 ts_tablet_manager.cc:1397] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:21.774291 26328 raft_consensus.cc:348] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:21.774366 26328 raft_consensus.cc:374] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:21.774387 26328 raft_consensus.cc:729] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06631da6424b44c2925f39a85ee8b4ee, State: Initialized, Role: FOLLOWER
I20260810 23:03:21.774344 26327 raft_consensus.cc:348] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:21.774413 26327 raft_consensus.cc:374] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:21.774444 26327 raft_consensus.cc:729] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0454b06429984d8585357a1f1a5c7d95, State: Initialized, Role: FOLLOWER
I20260810 23:03:21.774436 26328 consensus_queue.cc:260] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:21.774498 26327 consensus_queue.cc:260] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:21.774540 26328 ts_tablet_manager.cc:1428] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:21.774603 26327 ts_tablet_manager.cc:1428] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:21.774639 26334 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:21.774679 26333 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:21.774735 26334 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:21.774756 26333 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:22.116711 26333 raft_consensus.cc:482] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:22.116803 26333 raft_consensus.cc:504] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.117003 26333 leader_election.cc:288] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231), 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515)
I20260810 23:03:22.117264 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06918c3f4de424fa020d7d301075ee9" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0454b06429984d8585357a1f1a5c7d95" is_pre_election: true
I20260810 23:03:22.117266 26087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06918c3f4de424fa020d7d301075ee9" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca" is_pre_election: true
I20260810 23:03:22.117357 26244 raft_consensus.cc:2454] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06631da6424b44c2925f39a85ee8b4ee in term 0.
I20260810 23:03:22.117378 26087 raft_consensus.cc:2454] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06631da6424b44c2925f39a85ee8b4ee in term 0.
I20260810 23:03:22.117530 25862 leader_election.cc:302] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0454b06429984d8585357a1f1a5c7d95, 06631da6424b44c2925f39a85ee8b4ee; no voters:
I20260810 23:03:22.117638 26333 raft_consensus.cc:2781] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:22.117690 26333 raft_consensus.cc:482] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:22.117725 26333 raft_consensus.cc:3037] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:22.118405 26333 raft_consensus.cc:504] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.118553 26333 leader_election.cc:288] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 election: Requested vote from peers 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231), 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515)
I20260810 23:03:22.118721 26087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06918c3f4de424fa020d7d301075ee9" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca"
I20260810 23:03:22.118760 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e06918c3f4de424fa020d7d301075ee9" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0454b06429984d8585357a1f1a5c7d95"
I20260810 23:03:22.118816 26087 raft_consensus.cc:3037] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:22.118826 26244 raft_consensus.cc:3037] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:22.119505 26087 raft_consensus.cc:2454] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06631da6424b44c2925f39a85ee8b4ee in term 1.
I20260810 23:03:22.119542 26244 raft_consensus.cc:2454] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06631da6424b44c2925f39a85ee8b4ee in term 1.
I20260810 23:03:22.119707 25863 leader_election.cc:302] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06631da6424b44c2925f39a85ee8b4ee, 38b8feaefef64d238daad3f374cbb9ca; no voters:
I20260810 23:03:22.119805 26333 raft_consensus.cc:2781] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:22.119937 26333 raft_consensus.cc:686] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [term 1 LEADER]: Becoming Leader. State: Replica: 06631da6424b44c2925f39a85ee8b4ee, State: Running, Role: LEADER
I20260810 23:03:22.120052 26333 consensus_queue.cc:237] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.120467 26351 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: TxnStatusTablet state changed. Reason: New leader 06631da6424b44c2925f39a85ee8b4ee. Latest consensus state: current_term: 1 leader_uuid: "06631da6424b44c2925f39a85ee8b4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:22.120572 26351 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:03:22.120750 26353 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:03:22.120867 26353 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:03:22.120889 25757 catalog_manager.cc:5495] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65). New cstate: current_term: 1 leader_uuid: "06631da6424b44c2925f39a85ee8b4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } health_report { overall_health: UNKNOWN } } }
I20260810 23:03:22.127491 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:22.124798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.23.56.65:36289)
0810 23:03:22.124840 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:22.124842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:22.124892 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:03:22.124981 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:22.124981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:22.125003 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:22.125059 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:22.125203 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.125657 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.126739 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.126910 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.126959 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:22.127214 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:22.127297 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:22.127352 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:22.127426 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 23:03:22.127441 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:03:22.127562 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:22.124755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:36289 (local address 127.0.0.1:38678)
0810 23:03:22.124840 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:22.124851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:22.124906 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:22.125084 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:22.125086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:22.125087 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:22.125139 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:22.125142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.125729 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.125730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:22.126660 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:22.126663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.126984 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.126985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:22.127067 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:22.127090 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:22.127357 (+ 267us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:22.127366 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 23:03:22.127423 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 23:03:22.127477 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:03:22.129268 26244 raft_consensus.cc:1261] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Refusing update from remote peer 06631da6424b44c2925f39a85ee8b4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:22.129524 26087 raft_consensus.cc:1261] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Refusing update from remote peer 06631da6424b44c2925f39a85ee8b4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:22.129514 26351 consensus_queue.cc:1035] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:22.129812 26351 consensus_queue.cc:1035] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:22.130631 26334 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: TxnStatusTablet state changed. Reason: New leader 06631da6424b44c2925f39a85ee8b4ee. Latest consensus state: current_term: 1 leader_uuid: "06631da6424b44c2925f39a85ee8b4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:22.130718 26334 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:22.131985 26333 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06631da6424b44c2925f39a85ee8b4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:22.132078 26333 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:03:22.132125 26333 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "06631da6424b44c2925f39a85ee8b4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:22.132171 26333 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 06631da6424b44c2925f39a85ee8b4ee: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:03:22.132121 26334 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06631da6424b44c2925f39a85ee8b4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:22.132205 26334 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 0454b06429984d8585357a1f1a5c7d95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:22.132227 26356 mvcc.cc:205] Tried to move back new op lower bound from 7317106696719175680 to 7317106696684355584. Current Snapshot: MvccSnapshot[applied={T|T < 7317106696719175680}]
I20260810 23:03:22.132827 26331 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: TxnStatusTablet state changed. Reason: New leader 06631da6424b44c2925f39a85ee8b4ee. Latest consensus state: current_term: 1 leader_uuid: "06631da6424b44c2925f39a85ee8b4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:22.132920 26331 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:22.132963 26331 tablet_replica.cc:439] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "06631da6424b44c2925f39a85ee8b4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } } }
I20260810 23:03:22.133011 26331 tablet_replica.cc:442] T e06918c3f4de424fa020d7d301075ee9 P 38b8feaefef64d238daad3f374cbb9ca: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:22.133517 26359 mvcc.cc:205] Tried to move back new op lower bound from 7317106696719175680 to 7317106696684355584. Current Snapshot: MvccSnapshot[applied={T|T < 7317106696719175680 or (T in {7317106696719175680})}]
I20260810 23:03:22.137607 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:22.134657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42810 (local address 127.23.56.67:40515)
0810 23:03:22.134694 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:22.134696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:22.134822 (+ 126us) server_negotiation.cc:407] Connection header received
0810 23:03:22.134877 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:22.134878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:22.134896 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:22.134973 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:22.135074 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.135888 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.136691 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.137002 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.137076 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:22.137252 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:22.137387 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:22.137474 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:22.137510 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 23:03:22.137536 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:03:22.142699 26321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:22.140090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:53189 (local address 127.23.56.65:36289)
0810 23:03:22.140137 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:22.140139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:22.140302 (+ 163us) server_negotiation.cc:407] Connection header received
0810 23:03:22.140377 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:22.140378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:22.140387 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:22.140428 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:22.140571 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.141011 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.142116 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.142286 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.142335 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:22.142521 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:22.142545 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:22.142582 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:22.142632 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 23:03:22.142647 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:03:22.142753 26370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:22.140019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:36289 (local address 127.23.56.67:53189)
0810 23:03:22.140214 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:22.140224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:22.140309 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:22.140445 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:22.140446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:22.140448 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:22.140517 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:22.140518 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.141080 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.141082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:22.142037 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:22.142040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.142350 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.142351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:22.142415 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:22.142429 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:22.142583 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:22.142584 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 23:03:22.142631 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:03:22.142669 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":93,"threads_started":1}
I20260810 23:03:22.145238 26087 raft_consensus.cc:1261] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Refusing update from remote peer 0454b06429984d8585357a1f1a5c7d95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:22.145455 25935 raft_consensus.cc:1261] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Refusing update from remote peer 0454b06429984d8585357a1f1a5c7d95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:22.145535 26340 consensus_queue.cc:1035] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:22.145684 26340 consensus_queue.cc:1035] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:22.147181 26357 mvcc.cc:205] Tried to move back new op lower bound from 7317106696785207296 to 7317106694912143360. Current Snapshot: MvccSnapshot[applied={T|T < 7317106696785207296}]
I20260810 23:03:22.155162 26323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:22.151842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:53457 (local address 127.23.56.67:40515)
0810 23:03:22.151883 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:22.151885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:22.152116 (+ 231us) server_negotiation.cc:407] Connection header received
0810 23:03:22.152140 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:22.152141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:22.152153 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:22.152213 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:22.152322 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.153083 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.154294 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.154601 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.154675 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:22.154889 (+ 214us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:22.154922 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:22.154976 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:22.155039 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 23:03:22.155060 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:03:22.155225 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:22.151812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:40515 (local address 127.23.56.65:53457)
0810 23:03:22.151976 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:22.151987 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:22.152051 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:22.152218 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:22.152219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:22.152221 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:22.152276 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:22.152277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.153149 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.153150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:22.154183 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:22.154187 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:22.154705 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:22.154706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:22.154757 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:22.154773 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:22.154990 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:22.154991 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 23:03:22.155025 (+ 34us) client_negotiation.cc:240] Negotiation successful
0810 23:03:22.155063 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":98,"threads_started":1}
I20260810 23:03:22.164192 25758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:22.164304 25758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:22.165061 25758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: Sending DeleteTablet for 3 replicas of tablet a0835c9444554d039476e0c7d7259ee8
I20260810 23:03:22.165354 26066 tablet_service.cc:1514] Processing DeleteTablet for tablet a0835c9444554d039476e0c7d7259ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:22 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:33000
I20260810 23:03:22.165616 26379 tablet_replica.cc:330] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: stopping tablet replica
I20260810 23:03:22.165727 26379 raft_consensus.cc:2229] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:22.165778 26379 raft_consensus.cc:2258] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:22.165849 26228 tablet_service.cc:1514] Processing DeleteTablet for tablet a0835c9444554d039476e0c7d7259ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:22 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:42800
I20260810 23:03:22.166112 26380 tablet_replica.cc:330] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: stopping tablet replica
I20260810 23:03:22.166145 25913 tablet_service.cc:1514] Processing DeleteTablet for tablet a0835c9444554d039476e0c7d7259ee8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:22 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:38668
I20260810 23:03:22.166204 26380 raft_consensus.cc:2229] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:22.166333 26380 raft_consensus.cc:2258] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:22.166652 25758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:03:22.166726 26380 ts_tablet_manager.cc:1905] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:03:22.166786 25758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:22.168133 26066 tablet_service.cc:1467] Processing CreateTablet for tablet 22769d8adf1e48a69af0eae5cd3cfa34 (DEFAULT_TABLE table=test-table [id=891404ab65b6460d8c6bb31b114ceb33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:22.168256 26066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22769d8adf1e48a69af0eae5cd3cfa34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:22.168659 26380 ts_tablet_manager.cc:1918] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:22.168725 26380 log.cc:1196] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/a0835c9444554d039476e0c7d7259ee8
I20260810 23:03:22.168994 26380 ts_tablet_manager.cc:1939] T a0835c9444554d039476e0c7d7259ee8 P 0454b06429984d8585357a1f1a5c7d95: Deleting consensus metadata
I20260810 23:03:22.169637 26325 tablet_bootstrap.cc:492] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: Bootstrap starting.
I20260810 23:03:22.170042 26325 tablet_bootstrap.cc:654] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:22.170104 25740 catalog_manager.cc:4841] TS 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515): tablet a0835c9444554d039476e0c7d7259ee8 (table test-table [id=336dd39259e244ceafe46a909df88563]) successfully deleted
I20260810 23:03:22.170372 26228 tablet_service.cc:1467] Processing CreateTablet for tablet 22769d8adf1e48a69af0eae5cd3cfa34 (DEFAULT_TABLE table=test-table [id=891404ab65b6460d8c6bb31b114ceb33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:22.170475 26228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22769d8adf1e48a69af0eae5cd3cfa34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:22.171242 25913 tablet_service.cc:1467] Processing CreateTablet for tablet 22769d8adf1e48a69af0eae5cd3cfa34 (DEFAULT_TABLE table=test-table [id=891404ab65b6460d8c6bb31b114ceb33]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:22.171360 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22769d8adf1e48a69af0eae5cd3cfa34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:22.171723 26327 tablet_bootstrap.cc:492] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: Bootstrap starting.
I20260810 23:03:22.172142 26327 tablet_bootstrap.cc:654] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:22.172658 26328 tablet_bootstrap.cc:492] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: Bootstrap starting.
I20260810 23:03:22.173085 26379 ts_tablet_manager.cc:1905] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:22.173225 26328 tablet_bootstrap.cc:654] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:22.173450 26327 tablet_bootstrap.cc:492] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: No bootstrap required, opened a new log
I20260810 23:03:22.173504 26327 ts_tablet_manager.cc:1397] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:03:22.173653 26327 raft_consensus.cc:348] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.173719 26327 raft_consensus.cc:374] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:22.173740 26327 raft_consensus.cc:729] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0454b06429984d8585357a1f1a5c7d95, State: Initialized, Role: FOLLOWER
I20260810 23:03:22.173789 26327 consensus_queue.cc:260] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.173882 26327 ts_tablet_manager.cc:1428] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:22.174822 26381 tablet_replica.cc:330] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: stopping tablet replica
I20260810 23:03:22.174908 26379 ts_tablet_manager.cc:1918] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:22.175005 26381 raft_consensus.cc:2229] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:22.175010 26379 log.cc:1196] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/a0835c9444554d039476e0c7d7259ee8
I20260810 23:03:22.175088 26381 raft_consensus.cc:2258] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:22.175333 26379 ts_tablet_manager.cc:1939] T a0835c9444554d039476e0c7d7259ee8 P 38b8feaefef64d238daad3f374cbb9ca: Deleting consensus metadata
I20260810 23:03:22.175601 26381 ts_tablet_manager.cc:1905] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:22.175617 26325 tablet_bootstrap.cc:492] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: No bootstrap required, opened a new log
I20260810 23:03:22.175662 26325 ts_tablet_manager.cc:1397] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260810 23:03:22.175808 26325 raft_consensus.cc:348] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.175820 25741 catalog_manager.cc:4841] TS 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231): tablet a0835c9444554d039476e0c7d7259ee8 (table test-table [id=336dd39259e244ceafe46a909df88563]) successfully deleted
I20260810 23:03:22.175876 26325 raft_consensus.cc:374] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:22.175897 26325 raft_consensus.cc:729] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b8feaefef64d238daad3f374cbb9ca, State: Initialized, Role: FOLLOWER
I20260810 23:03:22.175962 26325 consensus_queue.cc:260] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.176055 26325 ts_tablet_manager.cc:1428] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:22.178421 26381 ts_tablet_manager.cc:1918] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:22.178521 26381 log.cc:1196] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/a0835c9444554d039476e0c7d7259ee8
I20260810 23:03:22.178800 26381 ts_tablet_manager.cc:1939] T a0835c9444554d039476e0c7d7259ee8 P 06631da6424b44c2925f39a85ee8b4ee: Deleting consensus metadata
I20260810 23:03:22.178975 26328 tablet_bootstrap.cc:492] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: No bootstrap required, opened a new log
I20260810 23:03:22.179044 26328 ts_tablet_manager.cc:1397] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.001s
I20260810 23:03:22.179219 26328 raft_consensus.cc:348] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.179304 26328 raft_consensus.cc:374] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:22.179332 26328 raft_consensus.cc:729] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06631da6424b44c2925f39a85ee8b4ee, State: Initialized, Role: FOLLOWER
I20260810 23:03:22.179385 26328 consensus_queue.cc:260] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.179497 26328 ts_tablet_manager.cc:1428] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:22.179590 25740 catalog_manager.cc:4841] TS 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289): tablet a0835c9444554d039476e0c7d7259ee8 (table test-table [id=336dd39259e244ceafe46a909df88563]) successfully deleted
I20260810 23:03:22.319483 26376 raft_consensus.cc:482] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:22.319571 26376 raft_consensus.cc:504] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.319718 26376 leader_election.cc:288] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289), 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231)
I20260810 23:03:22.319900 25935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22769d8adf1e48a69af0eae5cd3cfa34" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06631da6424b44c2925f39a85ee8b4ee" is_pre_election: true
I20260810 23:03:22.319967 26087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22769d8adf1e48a69af0eae5cd3cfa34" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca" is_pre_election: true
I20260810 23:03:22.320051 26087 raft_consensus.cc:2454] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 0.
I20260810 23:03:22.320048 25935 raft_consensus.cc:2454] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 0.
I20260810 23:03:22.320222 26173 leader_election.cc:302] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0454b06429984d8585357a1f1a5c7d95, 06631da6424b44c2925f39a85ee8b4ee; no voters:
I20260810 23:03:22.320320 26376 raft_consensus.cc:2781] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:22.320360 26376 raft_consensus.cc:482] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:22.320389 26376 raft_consensus.cc:3037] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:22.321074 26376 raft_consensus.cc:504] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.321213 26376 leader_election.cc:288] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 election: Requested vote from peers 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289), 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231)
I20260810 23:03:22.321379 25935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22769d8adf1e48a69af0eae5cd3cfa34" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06631da6424b44c2925f39a85ee8b4ee"
I20260810 23:03:22.321376 26087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22769d8adf1e48a69af0eae5cd3cfa34" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca"
I20260810 23:03:22.321457 25935 raft_consensus.cc:3037] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:22.321457 26087 raft_consensus.cc:3037] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:22.322227 26087 raft_consensus.cc:2454] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 1.
I20260810 23:03:22.322242 25935 raft_consensus.cc:2454] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 1.
I20260810 23:03:22.322407 26173 leader_election.cc:302] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0454b06429984d8585357a1f1a5c7d95, 06631da6424b44c2925f39a85ee8b4ee; no voters:
I20260810 23:03:22.322503 26376 raft_consensus.cc:2781] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:22.322598 26376 raft_consensus.cc:686] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 1 LEADER]: Becoming Leader. State: Replica: 0454b06429984d8585357a1f1a5c7d95, State: Running, Role: LEADER
I20260810 23:03:22.322667 26376 consensus_queue.cc:237] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:22.323230 25758 catalog_manager.cc:5495] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67). New cstate: current_term: 1 leader_uuid: "0454b06429984d8585357a1f1a5c7d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } health_report { overall_health: HEALTHY } } }
I20260810 23:03:22.358932 25935 raft_consensus.cc:1261] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Refusing update from remote peer 0454b06429984d8585357a1f1a5c7d95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:22.359090 26087 raft_consensus.cc:1261] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Refusing update from remote peer 0454b06429984d8585357a1f1a5c7d95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:22.359225 26334 consensus_queue.cc:1035] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:22.359352 26376 consensus_queue.cc:1035] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:03:22.364073 26357 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:03:23.881320 25758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:23.881438 25758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:23.882243 25758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: Sending DeleteTablet for 3 replicas of tablet 22769d8adf1e48a69af0eae5cd3cfa34
I20260810 23:03:23.882498 26066 tablet_service.cc:1514] Processing DeleteTablet for tablet 22769d8adf1e48a69af0eae5cd3cfa34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:23 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:33000
I20260810 23:03:23.882540 25913 tablet_service.cc:1514] Processing DeleteTablet for tablet 22769d8adf1e48a69af0eae5cd3cfa34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:23 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:38668
I20260810 23:03:23.882565 26228 tablet_service.cc:1514] Processing DeleteTablet for tablet 22769d8adf1e48a69af0eae5cd3cfa34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:23 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:42800
I20260810 23:03:23.882783 26414 tablet_replica.cc:330] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: stopping tablet replica
I20260810 23:03:23.882845 26414 raft_consensus.cc:2229] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:23.882879 26414 raft_consensus.cc:2258] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:23.882974 26413 tablet_replica.cc:330] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: stopping tablet replica
I20260810 23:03:23.883033 26413 raft_consensus.cc:2229] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:23.883064 26413 raft_consensus.cc:2258] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:23.883288 26414 ts_tablet_manager.cc:1905] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:23.884806 26413 ts_tablet_manager.cc:1905] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:23.884856 26414 ts_tablet_manager.cc:1918] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:03:23.884917 26414 log.cc:1196] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/22769d8adf1e48a69af0eae5cd3cfa34
I20260810 23:03:23.885165 26414 ts_tablet_manager.cc:1939] T 22769d8adf1e48a69af0eae5cd3cfa34 P 06631da6424b44c2925f39a85ee8b4ee: Deleting consensus metadata
I20260810 23:03:23.885464 25740 catalog_manager.cc:4841] TS 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289): tablet 22769d8adf1e48a69af0eae5cd3cfa34 (table test-table [id=891404ab65b6460d8c6bb31b114ceb33]) successfully deleted
I20260810 23:03:23.886380 26413 ts_tablet_manager.cc:1918] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:03:23.886440 26413 log.cc:1196] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/22769d8adf1e48a69af0eae5cd3cfa34
I20260810 23:03:23.886662 26413 ts_tablet_manager.cc:1939] T 22769d8adf1e48a69af0eae5cd3cfa34 P 38b8feaefef64d238daad3f374cbb9ca: Deleting consensus metadata
I20260810 23:03:23.886819 26415 tablet_replica.cc:330] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: stopping tablet replica
I20260810 23:03:23.886884 26415 raft_consensus.cc:2229] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:23.886946 26415 raft_consensus.cc:2258] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:23.887202 26415 ts_tablet_manager.cc:1905] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:23.887281 25741 catalog_manager.cc:4841] TS 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231): tablet 22769d8adf1e48a69af0eae5cd3cfa34 (table test-table [id=891404ab65b6460d8c6bb31b114ceb33]) successfully deleted
I20260810 23:03:23.888566 26415 ts_tablet_manager.cc:1918] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:03:23.888635 26415 log.cc:1196] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/22769d8adf1e48a69af0eae5cd3cfa34
I20260810 23:03:23.888895 26415 ts_tablet_manager.cc:1939] T 22769d8adf1e48a69af0eae5cd3cfa34 P 0454b06429984d8585357a1f1a5c7d95: Deleting consensus metadata
I20260810 23:03:23.889242 25740 catalog_manager.cc:4841] TS 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515): tablet 22769d8adf1e48a69af0eae5cd3cfa34 (table test-table [id=891404ab65b6460d8c6bb31b114ceb33]) successfully deleted
Aug 10 23:03:23 dist-test-slave-1xrh krb5kdc[25705](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:03:23 dist-test-slave-1xrh krb5kdc[25705](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:23.914232 26424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:23.900134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52906 (local address 127.23.56.126:38539)
0810 23:03:23.900284 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:23.900287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:23.900296 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:23.900393 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:23.900394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:23.900406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:23.900497 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:23.901043 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:23.901565 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:23.902292 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:23.902515 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:23.902600 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:23.911588 (+ 8988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:23.911590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:23.911591 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:23.911597 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:23.913037 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:23.913384 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:23.913385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:23.913386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:23.913449 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:23.913685 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:23.913686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:23.913686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:23.913805 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:23.913908 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:23.914034 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 23:03:23.914122 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260810 23:03:23.915814 25758 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:23.915969 25758 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:23.917340 26228 tablet_service.cc:1467] Processing CreateTablet for tablet f07b1029170244c1b00a899dce6892d2 (DEFAULT_TABLE table=test-table [id=a315be9740f5402aa3c857d0f707b481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:23.917325 25913 tablet_service.cc:1467] Processing CreateTablet for tablet f07b1029170244c1b00a899dce6892d2 (DEFAULT_TABLE table=test-table [id=a315be9740f5402aa3c857d0f707b481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:23.917331 26066 tablet_service.cc:1467] Processing CreateTablet for tablet f07b1029170244c1b00a899dce6892d2 (DEFAULT_TABLE table=test-table [id=a315be9740f5402aa3c857d0f707b481]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:23.917466 26228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07b1029170244c1b00a899dce6892d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:23.917466 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07b1029170244c1b00a899dce6892d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:23.917485 26066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f07b1029170244c1b00a899dce6892d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:23.918497 26425 tablet_bootstrap.cc:492] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: Bootstrap starting.
I20260810 23:03:23.918910 26426 tablet_bootstrap.cc:492] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: Bootstrap starting.
I20260810 23:03:23.918931 26425 tablet_bootstrap.cc:654] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:23.919217 26427 tablet_bootstrap.cc:492] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: Bootstrap starting.
I20260810 23:03:23.919302 26426 tablet_bootstrap.cc:654] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:23.919608 26425 tablet_bootstrap.cc:492] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: No bootstrap required, opened a new log
I20260810 23:03:23.919615 26427 tablet_bootstrap.cc:654] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:23.919660 26425 ts_tablet_manager.cc:1397] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:23.919812 26425 raft_consensus.cc:348] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:23.919903 26425 raft_consensus.cc:374] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:23.919945 26425 raft_consensus.cc:729] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b8feaefef64d238daad3f374cbb9ca, State: Initialized, Role: FOLLOWER
I20260810 23:03:23.920010 26425 consensus_queue.cc:260] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:23.920055 26426 tablet_bootstrap.cc:492] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: No bootstrap required, opened a new log
I20260810 23:03:23.920099 26426 ts_tablet_manager.cc:1397] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:23.920188 26425 ts_tablet_manager.cc:1428] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:23.920238 26426 raft_consensus.cc:348] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:23.920311 26426 raft_consensus.cc:374] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:23.920336 26426 raft_consensus.cc:729] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0454b06429984d8585357a1f1a5c7d95, State: Initialized, Role: FOLLOWER
I20260810 23:03:23.920403 26426 consensus_queue.cc:260] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:23.920513 26426 ts_tablet_manager.cc:1428] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:23.920786 26427 tablet_bootstrap.cc:492] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: No bootstrap required, opened a new log
I20260810 23:03:23.920845 26427 ts_tablet_manager.cc:1397] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:03:23.920964 26427 raft_consensus.cc:348] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:23.921056 26427 raft_consensus.cc:374] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:23.921085 26427 raft_consensus.cc:729] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06631da6424b44c2925f39a85ee8b4ee, State: Initialized, Role: FOLLOWER
I20260810 23:03:23.921159 26427 consensus_queue.cc:260] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:23.921283 26427 ts_tablet_manager.cc:1428] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:23.977839 26406 raft_consensus.cc:482] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:23.977959 26406 raft_consensus.cc:504] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:23.978184 26406 leader_election.cc:288] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289), 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231)
I20260810 23:03:23.978406 26087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07b1029170244c1b00a899dce6892d2" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca" is_pre_election: true
I20260810 23:03:23.978406 25935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07b1029170244c1b00a899dce6892d2" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06631da6424b44c2925f39a85ee8b4ee" is_pre_election: true
I20260810 23:03:23.978515 26087 raft_consensus.cc:2454] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 0.
I20260810 23:03:23.978518 25935 raft_consensus.cc:2454] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 0.
I20260810 23:03:23.978675 26174 leader_election.cc:302] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0454b06429984d8585357a1f1a5c7d95, 38b8feaefef64d238daad3f374cbb9ca; no voters:
I20260810 23:03:23.978804 26406 raft_consensus.cc:2781] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:23.978848 26406 raft_consensus.cc:482] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:23.978896 26406 raft_consensus.cc:3037] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:23.979748 26406 raft_consensus.cc:504] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:23.979899 26406 leader_election.cc:288] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 election: Requested vote from peers 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289), 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231)
I20260810 23:03:23.980090 26087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07b1029170244c1b00a899dce6892d2" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca"
I20260810 23:03:23.980178 26087 raft_consensus.cc:3037] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:23.980145 25935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f07b1029170244c1b00a899dce6892d2" candidate_uuid: "0454b06429984d8585357a1f1a5c7d95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06631da6424b44c2925f39a85ee8b4ee"
I20260810 23:03:23.980247 25935 raft_consensus.cc:3037] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:23.980814 26087 raft_consensus.cc:2454] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 1.
I20260810 23:03:23.980938 25935 raft_consensus.cc:2454] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0454b06429984d8585357a1f1a5c7d95 in term 1.
I20260810 23:03:23.980996 26174 leader_election.cc:302] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0454b06429984d8585357a1f1a5c7d95, 38b8feaefef64d238daad3f374cbb9ca; no voters:
I20260810 23:03:23.981091 26406 raft_consensus.cc:2781] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:23.981156 26406 raft_consensus.cc:686] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 1 LEADER]: Becoming Leader. State: Replica: 0454b06429984d8585357a1f1a5c7d95, State: Running, Role: LEADER
I20260810 23:03:23.981211 26406 consensus_queue.cc:237] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:23.981794 25758 catalog_manager.cc:5495] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67). New cstate: current_term: 1 leader_uuid: "0454b06429984d8585357a1f1a5c7d95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } health_report { overall_health: HEALTHY } } }
I20260810 23:03:23.997483 26436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:23.995409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.23.56.67:40515)
0810 23:03:23.995584 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:23.995586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:23.995593 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:03:23.995612 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:23.995612 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:23.995622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:23.995680 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:23.995791 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:23.996252 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:23.996955 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:23.997140 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:23.997188 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:23.997304 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:23.997328 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:23.997365 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:23.997411 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 23:03:23.997427 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":118,"threads_started":1}
I20260810 23:03:23.999439 25935 raft_consensus.cc:1261] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Refusing update from remote peer 0454b06429984d8585357a1f1a5c7d95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:23.999476 26087 raft_consensus.cc:1261] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Refusing update from remote peer 0454b06429984d8585357a1f1a5c7d95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:23.999655 26402 consensus_queue.cc:1035] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:23.999770 26406 consensus_queue.cc:1035] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:24.009512 25756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:24.009596 25756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:24.010187 25756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: Sending DeleteTablet for 3 replicas of tablet f07b1029170244c1b00a899dce6892d2
I20260810 23:03:24.010459 26228 tablet_service.cc:1514] Processing DeleteTablet for tablet f07b1029170244c1b00a899dce6892d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:24 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:42800
I20260810 23:03:24.010562 26415 tablet_replica.cc:330] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: stopping tablet replica
I20260810 23:03:24.010610 26415 raft_consensus.cc:2229] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:24.010666 26415 raft_consensus.cc:2258] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:24.010759 26066 tablet_service.cc:1514] Processing DeleteTablet for tablet f07b1029170244c1b00a899dce6892d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:24 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:33000
I20260810 23:03:24.010769 25756 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 23:03:24.010864 26413 tablet_replica.cc:330] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: stopping tablet replica
W20260810 23:03:24.010897 25756 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:24.010905 26413 raft_consensus.cc:2229] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:24.010941 26413 raft_consensus.cc:2258] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:24.011018 26415 ts_tablet_manager.cc:1905] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:24.011092 25913 tablet_service.cc:1514] Processing DeleteTablet for tablet f07b1029170244c1b00a899dce6892d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:24 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:38668
I20260810 23:03:24.011209 26414 tablet_replica.cc:330] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: stopping tablet replica
I20260810 23:03:24.011277 26414 raft_consensus.cc:2229] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:24.011317 26414 raft_consensus.cc:2258] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:24.011605 26414 ts_tablet_manager.cc:1905] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:24.012847 26415 ts_tablet_manager.cc:1918] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:24.012931 26415 log.cc:1196] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/f07b1029170244c1b00a899dce6892d2
I20260810 23:03:24.012931 26413 ts_tablet_manager.cc:1905] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:24.013206 26415 ts_tablet_manager.cc:1939] T f07b1029170244c1b00a899dce6892d2 P 0454b06429984d8585357a1f1a5c7d95: Deleting consensus metadata
I20260810 23:03:24.013240 26414 ts_tablet_manager.cc:1918] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:24.013305 26414 log.cc:1196] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/f07b1029170244c1b00a899dce6892d2
I20260810 23:03:24.013530 26414 ts_tablet_manager.cc:1939] T f07b1029170244c1b00a899dce6892d2 P 06631da6424b44c2925f39a85ee8b4ee: Deleting consensus metadata
I20260810 23:03:24.013587 25740 catalog_manager.cc:4841] TS 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515): tablet f07b1029170244c1b00a899dce6892d2 (table test-table [id=a315be9740f5402aa3c857d0f707b481]) successfully deleted
I20260810 23:03:24.014093 25740 catalog_manager.cc:4841] TS 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289): tablet f07b1029170244c1b00a899dce6892d2 (table test-table [id=a315be9740f5402aa3c857d0f707b481]) successfully deleted
I20260810 23:03:24.014446 26413 ts_tablet_manager.cc:1918] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:24.014530 26413 log.cc:1196] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/f07b1029170244c1b00a899dce6892d2
I20260810 23:03:24.014752 26413 ts_tablet_manager.cc:1939] T f07b1029170244c1b00a899dce6892d2 P 38b8feaefef64d238daad3f374cbb9ca: Deleting consensus metadata
I20260810 23:03:24.014977 25741 catalog_manager.cc:4841] TS 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231): tablet f07b1029170244c1b00a899dce6892d2 (table test-table [id=a315be9740f5402aa3c857d0f707b481]) successfully deleted
I20260810 23:03:24.015754 26228 tablet_service.cc:1467] Processing CreateTablet for tablet 399b1d29df7247c1a054bfce69238ccf (DEFAULT_TABLE table=test-table [id=38744570cbf8482e9cc774fec4c6c860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:24.015740 26066 tablet_service.cc:1467] Processing CreateTablet for tablet 399b1d29df7247c1a054bfce69238ccf (DEFAULT_TABLE table=test-table [id=38744570cbf8482e9cc774fec4c6c860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:24.015853 26066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399b1d29df7247c1a054bfce69238ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:24.015854 26228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399b1d29df7247c1a054bfce69238ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:24.016983 26425 tablet_bootstrap.cc:492] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: Bootstrap starting.
I20260810 23:03:24.017000 26426 tablet_bootstrap.cc:492] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: Bootstrap starting.
I20260810 23:03:24.017028 25913 tablet_service.cc:1467] Processing CreateTablet for tablet 399b1d29df7247c1a054bfce69238ccf (DEFAULT_TABLE table=test-table [id=38744570cbf8482e9cc774fec4c6c860]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:24.017120 25913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 399b1d29df7247c1a054bfce69238ccf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:24.017320 26426 tablet_bootstrap.cc:654] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:24.017416 26425 tablet_bootstrap.cc:654] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:24.017871 26426 tablet_bootstrap.cc:492] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: No bootstrap required, opened a new log
I20260810 23:03:24.017937 26426 ts_tablet_manager.cc:1397] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:24.018044 26426 raft_consensus.cc:348] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:24.018122 26426 raft_consensus.cc:374] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:24.018173 26426 raft_consensus.cc:729] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0454b06429984d8585357a1f1a5c7d95, State: Initialized, Role: FOLLOWER
I20260810 23:03:24.018225 26426 consensus_queue.cc:260] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:24.018322 26426 ts_tablet_manager.cc:1428] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:24.018337 26427 tablet_bootstrap.cc:492] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee: Bootstrap starting.
I20260810 23:03:24.018754 26425 tablet_bootstrap.cc:492] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: No bootstrap required, opened a new log
I20260810 23:03:24.018754 26427 tablet_bootstrap.cc:654] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:24.018786 26425 ts_tablet_manager.cc:1397] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:03:24.018891 26425 raft_consensus.cc:348] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:24.018966 26425 raft_consensus.cc:374] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:24.018990 26425 raft_consensus.cc:729] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b8feaefef64d238daad3f374cbb9ca, State: Initialized, Role: FOLLOWER
I20260810 23:03:24.019035 26425 consensus_queue.cc:260] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:24.019167 26425 ts_tablet_manager.cc:1428] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:24.019585 26427 tablet_bootstrap.cc:492] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee: No bootstrap required, opened a new log
I20260810 23:03:24.019663 26427 ts_tablet_manager.cc:1397] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:24.019773 26427 raft_consensus.cc:348] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:24.019843 26427 raft_consensus.cc:374] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:24.019867 26427 raft_consensus.cc:729] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06631da6424b44c2925f39a85ee8b4ee, State: Initialized, Role: FOLLOWER
I20260810 23:03:24.019917 26427 consensus_queue.cc:260] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:24.020021 26427 ts_tablet_manager.cc:1428] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:24.036044 26389 raft_consensus.cc:482] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:24.036108 26389 raft_consensus.cc:504] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:24.036245 26389 leader_election.cc:288] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231), 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515)
I20260810 23:03:24.036401 26087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b1d29df7247c1a054bfce69238ccf" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca" is_pre_election: true
I20260810 23:03:24.036422 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b1d29df7247c1a054bfce69238ccf" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0454b06429984d8585357a1f1a5c7d95" is_pre_election: true
I20260810 23:03:24.036477 26087 raft_consensus.cc:2454] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06631da6424b44c2925f39a85ee8b4ee in term 0.
I20260810 23:03:24.036535 26244 raft_consensus.cc:2454] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 06631da6424b44c2925f39a85ee8b4ee in term 0.
I20260810 23:03:24.036682 25863 leader_election.cc:302] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06631da6424b44c2925f39a85ee8b4ee, 38b8feaefef64d238daad3f374cbb9ca; no voters:
I20260810 23:03:24.036794 26389 raft_consensus.cc:2781] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:24.036859 26389 raft_consensus.cc:482] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:24.036887 26389 raft_consensus.cc:3037] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:24.037662 26389 raft_consensus.cc:504] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:24.037799 26389 leader_election.cc:288] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 election: Requested vote from peers 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231), 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515)
I20260810 23:03:24.037923 26244 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b1d29df7247c1a054bfce69238ccf" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0454b06429984d8585357a1f1a5c7d95"
I20260810 23:03:24.037943 26087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "399b1d29df7247c1a054bfce69238ccf" candidate_uuid: "06631da6424b44c2925f39a85ee8b4ee" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38b8feaefef64d238daad3f374cbb9ca"
I20260810 23:03:24.037993 26244 raft_consensus.cc:3037] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:24.038010 26087 raft_consensus.cc:3037] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:24.038671 26087 raft_consensus.cc:2454] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06631da6424b44c2925f39a85ee8b4ee in term 1.
I20260810 23:03:24.038690 26244 raft_consensus.cc:2454] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 06631da6424b44c2925f39a85ee8b4ee in term 1.
I20260810 23:03:24.038825 25863 leader_election.cc:302] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06631da6424b44c2925f39a85ee8b4ee, 38b8feaefef64d238daad3f374cbb9ca; no voters:
I20260810 23:03:24.038944 26389 raft_consensus.cc:2781] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:24.039021 26389 raft_consensus.cc:686] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [term 1 LEADER]: Becoming Leader. State: Replica: 06631da6424b44c2925f39a85ee8b4ee, State: Running, Role: LEADER
I20260810 23:03:24.039073 26389 consensus_queue.cc:237] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } }
I20260810 23:03:24.039654 25756 catalog_manager.cc:5495] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee reported cstate change: term changed from 0 to 1, leader changed from <none> to 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65). New cstate: current_term: 1 leader_uuid: "06631da6424b44c2925f39a85ee8b4ee" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06631da6424b44c2925f39a85ee8b4ee" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 } health_report { overall_health: UNKNOWN } } }
I20260810 23:03:24.048748 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:24.046640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.23.56.65:36289)
0810 23:03:24.046848 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:24.046850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:24.046859 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:24.046882 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:24.046883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:24.046893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:24.046964 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:24.047079 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:24.047602 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:24.048259 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:24.048440 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:24.048491 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:24.048595 (+ 104us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:24.048618 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:24.048650 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:24.048679 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 23:03:24.048694 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":133,"threads_started":1}
I20260810 23:03:24.052963 26445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:24.050749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:43381 (local address 127.23.56.65:36289)
0810 23:03:24.050781 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:24.050783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:24.050871 (+ 88us) server_negotiation.cc:407] Connection header received
0810 23:03:24.050908 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:24.050909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:24.050916 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:24.050959 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:24.051051 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:24.051536 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:24.052230 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:24.052421 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:24.052501 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:24.052688 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:24.052733 (+ 45us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:24.052806 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:24.052865 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 23:03:24.052881 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260810 23:03:24.052973 26449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:24.050687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:36289 (local address 127.23.56.65:43381)
0810 23:03:24.050804 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:24.050814 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:24.050870 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:24.050971 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:24.050972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:24.050973 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:24.051009 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:24.051009 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:24.051624 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:24.051625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:24.052146 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:24.052148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:24.052508 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:24.052509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:24.052577 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:24.052599 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:24.052811 (+ 212us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:24.052812 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 23:03:24.052859 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:03:24.052898 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":75,"threads_started":1}
I20260810 23:03:24.054560 26244 raft_consensus.cc:1261] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Refusing update from remote peer 06631da6424b44c2925f39a85ee8b4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:24.054569 26087 raft_consensus.cc:1261] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Refusing update from remote peer 06631da6424b44c2925f39a85ee8b4ee: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:24.054745 26392 consensus_queue.cc:1035] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b8feaefef64d238daad3f374cbb9ca" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:24.054893 26392 consensus_queue.cc:1035] T 399b1d29df7247c1a054bfce69238ccf P 06631da6424b44c2925f39a85ee8b4ee [LEADER]: Connected to new peer: Peer: permanent_uuid: "0454b06429984d8585357a1f1a5c7d95" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 23:03:24.058863 26390 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 23:03:25.576679 25756 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:25.576781 25756 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:25.577570 25756 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a21c6e3b79e4d95b0d7964011168dfd: Sending DeleteTablet for 3 replicas of tablet 399b1d29df7247c1a054bfce69238ccf
I20260810 23:03:25.577822 26066 tablet_service.cc:1514] Processing DeleteTablet for tablet 399b1d29df7247c1a054bfce69238ccf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:25 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:33000
I20260810 23:03:25.577831 26228 tablet_service.cc:1514] Processing DeleteTablet for tablet 399b1d29df7247c1a054bfce69238ccf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:25 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:42800
I20260810 23:03:25.577973 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 25842
I20260810 23:03:25.580453 26475 tablet_replica.cc:330] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: stopping tablet replica
I20260810 23:03:25.580510 26475 raft_consensus.cc:2229] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:25.580536 26475 raft_consensus.cc:2258] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:25.580643 26474 tablet_replica.cc:330] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: stopping tablet replica
I20260810 23:03:25.580698 26474 raft_consensus.cc:2229] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:25.580727 26474 raft_consensus.cc:2258] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:25.580803 26475 ts_tablet_manager.cc:1905] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:25.580947 26474 ts_tablet_manager.cc:1905] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:25.582496 26474 ts_tablet_manager.cc:1918] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:03:25.582571 26474 log.cc:1196] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/399b1d29df7247c1a054bfce69238ccf
I20260810 23:03:25.582763 26475 ts_tablet_manager.cc:1918] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 23:03:25.582834 26475 log.cc:1196] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/399b1d29df7247c1a054bfce69238ccf
I20260810 23:03:25.582872 26474 ts_tablet_manager.cc:1939] T 399b1d29df7247c1a054bfce69238ccf P 0454b06429984d8585357a1f1a5c7d95: Deleting consensus metadata
I20260810 23:03:25.583161 26475 ts_tablet_manager.cc:1939] T 399b1d29df7247c1a054bfce69238ccf P 38b8feaefef64d238daad3f374cbb9ca: Deleting consensus metadata
I20260810 23:03:25.583240 25740 catalog_manager.cc:4841] TS 0454b06429984d8585357a1f1a5c7d95 (127.23.56.67:40515): tablet 399b1d29df7247c1a054bfce69238ccf (table test-table [id=38744570cbf8482e9cc774fec4c6c860]) successfully deleted
I20260810 23:03:25.583446 25741 catalog_manager.cc:4841] TS 38b8feaefef64d238daad3f374cbb9ca (127.23.56.66:36231): tablet 399b1d29df7247c1a054bfce69238ccf (table test-table [id=38744570cbf8482e9cc774fec4c6c860]) successfully deleted
W20260810 23:03:25.588201 25740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:36289 (error 108)
I20260810 23:03:25.588487 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 25996
I20260810 23:03:25.590096 26477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:25.588602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:36289 (local address 127.0.0.1:38688)
0810 23:03:25.590016 (+ 1414us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:25.590061 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1394,"thread_start_us":60,"threads_started":1}
W20260810 23:03:25.590204 25740 catalog_manager.cc:4584] TS 06631da6424b44c2925f39a85ee8b4ee (127.23.56.65:36289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 399b1d29df7247c1a054bfce69238ccf: Network error: Client connection negotiation failed: client connection to 127.23.56.65:36289: connect: Connection refused (error 111)
I20260810 23:03:25.595568 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 26154
I20260810 23:03:25.602254 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 25719
2026-08-10T23:03:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6211 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:25 dist-test-slave-1xrh krb5kdc[26480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:25 dist-test-slave-1xrh krb5kdc[26480](info): set up 2 sockets
Aug 10 23:03:25 dist-test-slave-1xrh krb5kdc[26480](info): commencing operation
krb5kdc: starting...
W20260810 23:03:26.703787 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:26 dist-test-slave-1xrh krb5kdc[26480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:26.822289 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:35959
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:35959
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:26.903859 26494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:26.904080 26494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:26.904120 26494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:26.905267 26494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:26.905339 26494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:26.905366 26494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:26.905404 26494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:26.905435 26494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:26.906895 26494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:35959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:35959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:26.907197 26494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:26.907424 26494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:26.912559 26499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:26.912547 26500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:26.912658 26494 server_base.cc:1034] running on GCE node
W20260810 23:03:26.912546 26502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:26.913038 26494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:26.913311 26494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:26.914448 26494 hybrid_clock.cc:648] HybridClock initialized: now 1786403006914431 us; error 39 us; skew 500 ppm
Aug 10 23:03:26 dist-test-slave-1xrh krb5kdc[26480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403006, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:26.916895 26494 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:26.917311 26494 webserver.cc:460] Webserver started at http://127.23.56.126:37345/ using document root <none> and password file <none>
I20260810 23:03:26.917553 26494 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:26.917619 26494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:26.917766 26494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:26.918816 26494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "20b7ac30eafc4e1ebbce49a227700c66"
format_stamp: "Formatted at 2026-08-10 23:03:26 on dist-test-slave-1xrh"
server_key: "01ae40705ebc94f51bca67ae953c338c"
server_key_iv: "9fbdedc83907a4307078db025c2072f7"
server_key_version: "encryptionkey@0"
I20260810 23:03:26.919217 26494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "20b7ac30eafc4e1ebbce49a227700c66"
format_stamp: "Formatted at 2026-08-10 23:03:26 on dist-test-slave-1xrh"
server_key: "01ae40705ebc94f51bca67ae953c338c"
server_key_iv: "9fbdedc83907a4307078db025c2072f7"
server_key_version: "encryptionkey@0"
I20260810 23:03:26.920542 26494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:26.921217 26509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:26.921382 26494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:26.921478 26494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "20b7ac30eafc4e1ebbce49a227700c66"
format_stamp: "Formatted at 2026-08-10 23:03:26 on dist-test-slave-1xrh"
server_key: "01ae40705ebc94f51bca67ae953c338c"
server_key_iv: "9fbdedc83907a4307078db025c2072f7"
server_key_version: "encryptionkey@0"
I20260810 23:03:26.921562 26494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:26.938894 26494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:26.941051 26494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:26.941237 26494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:26.945484 26494 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:35959
I20260810 23:03:26.945518 26566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:35959 every 8 connection(s)
I20260810 23:03:26.945880 26494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:26.946775 26567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:26.946786 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 26494
I20260810 23:03:26.946911 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:26.947161 23777 external_mini_cluster.cc:1428] Setting key 2b846a5a7496bedf31e04d84bf1619a6
I20260810 23:03:26.949930 26567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66: Bootstrap starting.
I20260810 23:03:26.950644 26567 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:26.950944 26567 log.cc:824] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:26.951683 26567 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66: No bootstrap required, opened a new log
Aug 10 23:03:26 dist-test-slave-1xrh krb5kdc[26480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:26.954288 26567 raft_consensus.cc:348] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b7ac30eafc4e1ebbce49a227700c66" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35959 } }
I20260810 23:03:26.954473 26567 raft_consensus.cc:374] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:26.954512 26567 raft_consensus.cc:729] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20b7ac30eafc4e1ebbce49a227700c66, State: Initialized, Role: FOLLOWER
I20260810 23:03:26.954653 26567 consensus_queue.cc:260] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b7ac30eafc4e1ebbce49a227700c66" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35959 } }
I20260810 23:03:26.954742 26567 raft_consensus.cc:388] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:26.954818 26567 raft_consensus.cc:482] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:26.954895 26567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:26.955474 26567 raft_consensus.cc:504] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b7ac30eafc4e1ebbce49a227700c66" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35959 } }
I20260810 23:03:26.955588 26567 leader_election.cc:302] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 20b7ac30eafc4e1ebbce49a227700c66; no voters:
I20260810 23:03:26.955758 26567 leader_election.cc:288] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:26.955804 26572 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:26.955950 26572 raft_consensus.cc:686] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [term 1 LEADER]: Becoming Leader. State: Replica: 20b7ac30eafc4e1ebbce49a227700c66, State: Running, Role: LEADER
I20260810 23:03:26.956012 26567 sys_catalog.cc:564] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:26.956086 26572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b7ac30eafc4e1ebbce49a227700c66" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35959 } }
I20260810 23:03:26.956496 26572 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 20b7ac30eafc4e1ebbce49a227700c66. Latest consensus state: current_term: 1 leader_uuid: "20b7ac30eafc4e1ebbce49a227700c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b7ac30eafc4e1ebbce49a227700c66" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35959 } } }
I20260810 23:03:26.956786 26572 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:26.956712 26574 sys_catalog.cc:455] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "20b7ac30eafc4e1ebbce49a227700c66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "20b7ac30eafc4e1ebbce49a227700c66" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35959 } } }
I20260810 23:03:26.957257 26574 sys_catalog.cc:458] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:26.957425 26583 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:26.958012 26583 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:26.958269 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:26.947682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41144 (local address 127.23.56.126:35959)
0810 23:03:26.947853 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:26.947861 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:26.947885 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:03:26.948316 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:26.948321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:26.948365 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:26.948477 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:26.949276 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:26.950207 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:26.951298 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:26.951748 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:26.951889 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:26.954345 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:26.954349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:26.954349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:26.954360 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:26.956419 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:26.956900 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:26.956902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:26.956903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:26.957012 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:26.957250 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:26.957251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:26.957251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:26.957676 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:26.957825 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:26.957923 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 23:03:26.958075 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260810 23:03:26.959486 26583 catalog_manager.cc:1347] Generated new cluster ID: fb0db5ce05964c3cba0dada18a638d1c
I20260810 23:03:26.959540 26583 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:26.970695 26583 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:26.971345 26583 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:26.974967 26583 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 20b7ac30eafc4e1ebbce49a227700c66: Generated new TSK 0
I20260810 23:03:26.975194 26583 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:27.005685 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:35959
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:27.088084 26596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:27.088301 26596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:27.088330 26596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:27.089425 26596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:27.089460 26596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:27.089489 26596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:27.090941 26596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:27.091194 26596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:27.091382 26596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:27.096329 26601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:27.096345 26604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:27.096391 26602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:27.096539 26596 server_base.cc:1034] running on GCE node
I20260810 23:03:27.096835 26596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:27.097093 26596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:27.098256 26596 hybrid_clock.cc:648] HybridClock initialized: now 1786403007098238 us; error 29 us; skew 500 ppm
Aug 10 23:03:27 dist-test-slave-1xrh krb5kdc[26480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403007, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:27.100916 26596 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:27.101357 26596 webserver.cc:460] Webserver started at http://127.23.56.65:35231/ using document root <none> and password file <none>
I20260810 23:03:27.101595 26596 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:27.101666 26596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:27.101819 26596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:27.102860 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "40f86575d76b4d569711102892102472"
format_stamp: "Formatted at 2026-08-10 23:03:27 on dist-test-slave-1xrh"
server_key: "db2d4ac612cef087f463246e334c9e1a"
server_key_iv: "d100945e68aea1f246b9e36418b146a5"
server_key_version: "encryptionkey@0"
I20260810 23:03:27.103276 26596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "40f86575d76b4d569711102892102472"
format_stamp: "Formatted at 2026-08-10 23:03:27 on dist-test-slave-1xrh"
server_key: "db2d4ac612cef087f463246e334c9e1a"
server_key_iv: "d100945e68aea1f246b9e36418b146a5"
server_key_version: "encryptionkey@0"
I20260810 23:03:27.104658 26596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:27.105391 26611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.105566 26596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:27.105666 26596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "40f86575d76b4d569711102892102472"
format_stamp: "Formatted at 2026-08-10 23:03:27 on dist-test-slave-1xrh"
server_key: "db2d4ac612cef087f463246e334c9e1a"
server_key_iv: "d100945e68aea1f246b9e36418b146a5"
server_key_version: "encryptionkey@0"
I20260810 23:03:27.105755 26596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:27.119654 26596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:27.121754 26596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:27.121908 26596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:27.122133 26596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:27.122430 26596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:27.122463 26596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.122504 26596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:27.122531 26596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.128517 26596 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:40085
I20260810 23:03:27.128608 26734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:40085 every 8 connection(s)
I20260810 23:03:27.128902 26596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:27.130771 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 26596
I20260810 23:03:27.130846 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:27.131031 23777 external_mini_cluster.cc:1428] Setting key f10760ec38e4daadde490e441966b430
Aug 10 23:03:27 dist-test-slave-1xrh krb5kdc[26480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403007, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:27.138833 26737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:27.129588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:35959 (local address 127.23.56.65:60513)
0810 23:03:27.129776 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:27.129811 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:27.130332 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:27.130584 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:27.130588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:27.131345 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:27.131636 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:27.131643 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.132313 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.132314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:27.133095 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:27.133097 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.133512 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.133513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:27.133641 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:27.134133 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:27.134141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:27.135647 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:27.137560 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:27.137567 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:27.137568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:27.137774 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:27.137984 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:27.137985 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:27.137985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:27.138087 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:27.138371 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:27.138376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:27.138483 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 23:03:27.138569 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:03:27.138681 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
I20260810 23:03:27.138851 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:27.129625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:60513 (local address 127.23.56.126:35959)
0810 23:03:27.129672 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:27.129674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:27.129892 (+ 218us) server_negotiation.cc:407] Connection header received
0810 23:03:27.130445 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:27.130446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:27.130459 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:27.130506 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:27.131718 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.132219 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.133198 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.133414 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.133495 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:27.135739 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:27.135740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:27.135741 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:27.135747 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:27.137473 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:27.137851 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:27.137853 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:27.137854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:27.137914 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:27.138152 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:27.138153 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:27.138153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:27.138302 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:27.138375 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:27.138619 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 23:03:27.138727 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:03:27.139575 26735 heartbeater.cc:347] Connected to a master server at 127.23.56.126:35959
I20260810 23:03:27.139688 26735 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:27.139860 26735 heartbeater.cc:511] Master 127.23.56.126:35959 requested a full tablet report, sending...
I20260810 23:03:27.140368 26526 ts_manager.cc:194] Registered new tserver with Master: 40f86575d76b4d569711102892102472 (127.23.56.65:40085)
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
I20260810 23:03:27.140974 26526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:60513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:27.158243 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:35959
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:27.239104 26740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:27.239317 26740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:27.239341 26740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:27.240478 26740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:27.240518 26740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:27.240567 26740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:27.241952 26740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26740
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:27.242182 26740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:27.242434 26740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:27.247401 26745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:27.247421 26749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:27.247401 26746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:27.247577 26740 server_base.cc:1034] running on GCE node
I20260810 23:03:27.247869 26740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:27.248129 26740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:27.249267 26740 hybrid_clock.cc:648] HybridClock initialized: now 1786403007249257 us; error 26 us; skew 500 ppm
Aug 10 23:03:27 dist-test-slave-1xrh krb5kdc[26480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403007, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:27.251677 26740 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:27.252143 26740 webserver.cc:460] Webserver started at http://127.23.56.66:37227/ using document root <none> and password file <none>
I20260810 23:03:27.252369 26740 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:27.252436 26740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:27.252611 26740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:27.253739 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "480f61fc4e2342f6be9cb6b785b8659a"
format_stamp: "Formatted at 2026-08-10 23:03:27 on dist-test-slave-1xrh"
server_key: "d696834a06c500e07d8066ee7f3fbcfc"
server_key_iv: "055b5dc6b30da31ad29405652371da60"
server_key_version: "encryptionkey@0"
I20260810 23:03:27.254089 26740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "480f61fc4e2342f6be9cb6b785b8659a"
format_stamp: "Formatted at 2026-08-10 23:03:27 on dist-test-slave-1xrh"
server_key: "d696834a06c500e07d8066ee7f3fbcfc"
server_key_iv: "055b5dc6b30da31ad29405652371da60"
server_key_version: "encryptionkey@0"
I20260810 23:03:27.255419 26740 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:03:27.256201 26756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.256371 26740 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:27.256466 26740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "480f61fc4e2342f6be9cb6b785b8659a"
format_stamp: "Formatted at 2026-08-10 23:03:27 on dist-test-slave-1xrh"
server_key: "d696834a06c500e07d8066ee7f3fbcfc"
server_key_iv: "055b5dc6b30da31ad29405652371da60"
server_key_version: "encryptionkey@0"
I20260810 23:03:27.256557 26740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:27.266942 26740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:27.268690 26740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:27.268847 26740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:27.269069 26740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:27.269358 26740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:27.269411 26740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.269452 26740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:27.269490 26740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.275295 26740 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:35731
I20260810 23:03:27.275348 26879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:35731 every 8 connection(s)
I20260810 23:03:27.275633 26740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
Aug 10 23:03:27 dist-test-slave-1xrh krb5kdc[26480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403007, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:27.283095 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 26740
I20260810 23:03:27.283263 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:27.283478 23777 external_mini_cluster.cc:1428] Setting key fcbca9602cef2aca57aa4cc4551596d6
I20260810 23:03:27.284587 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:27.276300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:40365 (local address 127.23.56.126:35959)
0810 23:03:27.276387 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:27.276390 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:27.276561 (+ 171us) server_negotiation.cc:407] Connection header received
0810 23:03:27.277041 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:27.277042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:27.277056 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:27.277109 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:27.278003 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.278511 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.279501 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.279726 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.279814 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:27.281903 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:27.281905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:27.281905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:27.281913 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:27.283070 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:27.283587 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:27.283588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:27.283588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:27.283654 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:27.283929 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:27.283930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:27.283930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:27.284049 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:27.284123 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:27.284399 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 23:03:27.284501 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:03:27.284766 26882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:27.276265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:35959 (local address 127.23.56.66:40365)
0810 23:03:27.276432 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:27.276472 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:27.276933 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:27.277168 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:27.277173 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:27.277709 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:27.277897 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:27.277901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.278618 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.278619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:27.279409 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:27.279411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.279827 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.279829 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:27.279948 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:27.280459 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:27.280466 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:27.281808 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:27.283175 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:27.283182 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:27.283185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:27.283490 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:27.283724 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:27.283725 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:27.283726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:27.283848 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:27.284126 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:27.284133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:27.284265 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 23:03:27.284366 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:03:27.284519 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260810 23:03:27.285251 26880 heartbeater.cc:347] Connected to a master server at 127.23.56.126:35959
I20260810 23:03:27.285357 26880 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:27.285552 26880 heartbeater.cc:511] Master 127.23.56.126:35959 requested a full tablet report, sending...
I20260810 23:03:27.285941 26526 ts_manager.cc:194] Registered new tserver with Master: 480f61fc4e2342f6be9cb6b785b8659a (127.23.56.66:35731)
I20260810 23:03:27.286358 26526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:40365
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:27.308082 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:35959
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:34757
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:27.396015 26886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:27.396265 26886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:27.396304 26886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:27.397524 26886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:27.397594 26886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:27.397642 26886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:27.399276 26886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:34757
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:35959
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:27.399567 26886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:27.399807 26886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:27.405109 26892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:27.405097 26893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:27.405234 26886 server_base.cc:1034] running on GCE node
W20260810 23:03:27.405077 26895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:27.405488 26886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:27.405728 26886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:27.406875 26886 hybrid_clock.cc:648] HybridClock initialized: now 1786403007406860 us; error 29 us; skew 500 ppm
Aug 10 23:03:27 dist-test-slave-1xrh krb5kdc[26480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403007, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:27.409494 26886 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:27.409909 26886 webserver.cc:460] Webserver started at http://127.23.56.67:33445/ using document root <none> and password file <none>
I20260810 23:03:27.410130 26886 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:27.410194 26886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:27.410373 26886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:27.411463 26886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "c05e645e65b34b83b9819e07acc8c339"
format_stamp: "Formatted at 2026-08-10 23:03:27 on dist-test-slave-1xrh"
server_key: "2a57fff83ff01966cf3d73d775d81ddd"
server_key_iv: "8cb184805e6b4f68b9a1a23286783df8"
server_key_version: "encryptionkey@0"
I20260810 23:03:27.411819 26886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "c05e645e65b34b83b9819e07acc8c339"
format_stamp: "Formatted at 2026-08-10 23:03:27 on dist-test-slave-1xrh"
server_key: "2a57fff83ff01966cf3d73d775d81ddd"
server_key_iv: "8cb184805e6b4f68b9a1a23286783df8"
server_key_version: "encryptionkey@0"
I20260810 23:03:27.413062 26886 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:27.413718 26903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.413864 26886 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.413959 26886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "c05e645e65b34b83b9819e07acc8c339"
format_stamp: "Formatted at 2026-08-10 23:03:27 on dist-test-slave-1xrh"
server_key: "2a57fff83ff01966cf3d73d775d81ddd"
server_key_iv: "8cb184805e6b4f68b9a1a23286783df8"
server_key_version: "encryptionkey@0"
I20260810 23:03:27.414036 26886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:27.431576 26886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:27.433794 26886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:27.433962 26886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:27.434185 26886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:27.434484 26886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:27.434538 26886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.434572 26886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:27.434602 26886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:27.440436 26886 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:34715
I20260810 23:03:27.440474 27024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:34715 every 8 connection(s)
I20260810 23:03:27.440810 26886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:27.444207 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 26886
I20260810 23:03:27.444295 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:27.444511 23777 external_mini_cluster.cc:1428] Setting key 007dd5d215da334ce51759fd5ff237f7
Aug 10 23:03:27 dist-test-slave-1xrh krb5kdc[26480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403007, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:27.449649 26570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:27.441538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:49589 (local address 127.23.56.126:35959)
0810 23:03:27.441581 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:27.441583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:27.441772 (+ 189us) server_negotiation.cc:407] Connection header received
0810 23:03:27.442247 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:27.442248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:27.442259 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:27.442305 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:27.443180 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.443661 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.444841 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.445159 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.445225 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:27.447381 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:27.447383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:27.447383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:27.447390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:27.448585 (+ 1195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:27.448915 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:27.448915 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:27.448916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:27.448960 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:27.449153 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:27.449153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:27.449154 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:27.449247 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:27.449294 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:27.449509 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 23:03:27.449586 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:03:27.449851 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:27.441510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:35959 (local address 127.23.56.67:49589)
0810 23:03:27.441657 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:27.441690 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:27.442152 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:27.442361 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:27.442370 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:27.442886 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:27.443061 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:27.443065 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.443745 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.443746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:27.444764 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:27.444767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:27.445246 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:27.445247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:27.445390 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:27.445967 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:27.445983 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:27.447301 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:27.448652 (+ 1351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:27.448658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:27.448659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:27.448853 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:27.449014 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:27.449014 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:27.449014 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:27.449102 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:27.449297 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:27.449301 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:27.449406 (+ 105us) client_negotiation.cc:769] Sending connection context
0810 23:03:27.449482 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:03:27.449610 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
I20260810 23:03:27.450373 27025 heartbeater.cc:347] Connected to a master server at 127.23.56.126:35959
I20260810 23:03:27.450464 27025 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:27.450598 27025 heartbeater.cc:511] Master 127.23.56.126:35959 requested a full tablet report, sending...
I20260810 23:03:27.450958 26526 ts_manager.cc:194] Registered new tserver with Master: c05e645e65b34b83b9819e07acc8c339 (127.23.56.67:34715)
I20260810 23:03:27.451386 26526 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:49589
I20260810 23:03:27.456173 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:03:27.469281 26570 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:27.468340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.23.56.126:35959)
0810 23:03:27.468398 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:27.468400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:27.468565 (+ 165us) server_negotiation.cc:407] Connection header received
0810 23:03:27.468650 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:27.468651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:27.468663 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:27.468728 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:27.469212 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41156: BlockingRecv error: recv got EOF from 127.0.0.1:41156 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:03:27.470105 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 26596
I20260810 23:03:27.479952 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 26740
I20260810 23:03:27.489249 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 26886
I20260810 23:03:27.495394 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 26494
2026-08-10T23:03:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1892 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:27 dist-test-slave-1xrh krb5kdc[27038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:27 dist-test-slave-1xrh krb5kdc[27038](info): set up 2 sockets
Aug 10 23:03:27 dist-test-slave-1xrh krb5kdc[27038](info): commencing operation
krb5kdc: starting...
W20260810 23:03:28.588142 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:28 dist-test-slave-1xrh krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:28.709915 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:42825
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:28.792289 27052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:28.792500 27052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:28.792524 27052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:28.793618 27052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:28.793654 27052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:28.793669 27052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:28.793682 27052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:28.793694 27052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:28.795197 27052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:42825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:28.795382 27052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:28.795554 27052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:28.800163 27058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:28.800189 27060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:28.800343 27052 server_base.cc:1034] running on GCE node
W20260810 23:03:28.800410 27057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:28.800669 27052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:28.800909 27052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:28.802074 27052 hybrid_clock.cc:648] HybridClock initialized: now 1786403008802053 us; error 36 us; skew 500 ppm
Aug 10 23:03:28 dist-test-slave-1xrh krb5kdc[27038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403008, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:28.804508 27052 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:28.804960 27052 webserver.cc:460] Webserver started at http://127.23.56.126:35089/ using document root <none> and password file <none>
I20260810 23:03:28.805184 27052 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:28.805253 27052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:28.805401 27052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:28.806407 27052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "608da53293434630b33daf5ea4c0d7e3"
format_stamp: "Formatted at 2026-08-10 23:03:28 on dist-test-slave-1xrh"
server_key: "3af8832d426239fef61362d652b2802d"
server_key_iv: "89a66ba7d8fdb3c85e0af3296c34617e"
server_key_version: "encryptionkey@0"
I20260810 23:03:28.806771 27052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "608da53293434630b33daf5ea4c0d7e3"
format_stamp: "Formatted at 2026-08-10 23:03:28 on dist-test-slave-1xrh"
server_key: "3af8832d426239fef61362d652b2802d"
server_key_iv: "89a66ba7d8fdb3c85e0af3296c34617e"
server_key_version: "encryptionkey@0"
I20260810 23:03:28.808153 27052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:28.808858 27067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:28.809060 27052 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:28.809176 27052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "608da53293434630b33daf5ea4c0d7e3"
format_stamp: "Formatted at 2026-08-10 23:03:28 on dist-test-slave-1xrh"
server_key: "3af8832d426239fef61362d652b2802d"
server_key_iv: "89a66ba7d8fdb3c85e0af3296c34617e"
server_key_version: "encryptionkey@0"
I20260810 23:03:28.809254 27052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:28.817195 27052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:28.818985 27052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:28.819204 27052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:28.823203 27052 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:42825
I20260810 23:03:28.823235 27122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:42825 every 8 connection(s)
I20260810 23:03:28.823793 27052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:28.824426 27123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:28.824453 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 27052
I20260810 23:03:28.824556 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:28.824723 23777 external_mini_cluster.cc:1428] Setting key 10d2a907684813d4dc3948fc7898aa07
I20260810 23:03:28.827412 27123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3: Bootstrap starting.
I20260810 23:03:28.828037 27123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:28.828313 27123 log.cc:824] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:28.828971 27123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3: No bootstrap required, opened a new log
I20260810 23:03:28.830832 27123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608da53293434630b33daf5ea4c0d7e3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42825 } }
I20260810 23:03:28.831012 27123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:28.831049 27123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 608da53293434630b33daf5ea4c0d7e3, State: Initialized, Role: FOLLOWER
I20260810 23:03:28.831216 27123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608da53293434630b33daf5ea4c0d7e3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42825 } }
I20260810 23:03:28.831300 27123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
Aug 10 23:03:28 dist-test-slave-1xrh krb5kdc[27038](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:28.831332 27123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:28.831393 27123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:28.832103 27123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608da53293434630b33daf5ea4c0d7e3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42825 } }
I20260810 23:03:28.832216 27123 leader_election.cc:302] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 608da53293434630b33daf5ea4c0d7e3; no voters:
I20260810 23:03:28.832382 27123 leader_election.cc:288] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:28.832422 27130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:28.832574 27130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [term 1 LEADER]: Becoming Leader. State: Replica: 608da53293434630b33daf5ea4c0d7e3, State: Running, Role: LEADER
I20260810 23:03:28.832665 27130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608da53293434630b33daf5ea4c0d7e3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42825 } }
I20260810 23:03:28.832669 27123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:28.833050 27131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "608da53293434630b33daf5ea4c0d7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608da53293434630b33daf5ea4c0d7e3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42825 } } }
I20260810 23:03:28.833133 27131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:28.833330 27132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 608da53293434630b33daf5ea4c0d7e3. Latest consensus state: current_term: 1 leader_uuid: "608da53293434630b33daf5ea4c0d7e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "608da53293434630b33daf5ea4c0d7e3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42825 } } }
I20260810 23:03:28.833410 27132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:28.833357 27136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:28.834616 27136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:28.835928 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:28.825294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.23.56.126:42825)
0810 23:03:28.825476 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:28.825495 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:28.825571 (+ 76us) server_negotiation.cc:407] Connection header received
0810 23:03:28.826064 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:28.826071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:28.826118 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:28.826265 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:28.826908 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:28.827881 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:28.828771 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:28.829216 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:28.829330 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:28.831882 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:28.831885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:28.831886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:28.831895 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:28.834026 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:28.834459 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:28.834462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:28.834463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:28.834567 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:28.834793 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:28.834794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:28.834794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:28.835097 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:28.835272 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:28.835591 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 23:03:28.835724 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260810 23:03:28.836351 27136 catalog_manager.cc:1347] Generated new cluster ID: df4eaff7a1d2465f90a61480f8faebde
I20260810 23:03:28.836407 27136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:28.837015 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:03:28.854068 27127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:28.849948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47254 (local address 127.23.56.126:42825)
0810 23:03:28.849990 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:28.849992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:28.850116 (+ 124us) server_negotiation.cc:407] Connection header received
0810 23:03:28.850165 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:28.850166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:28.850179 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:28.850254 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:28.850652 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:28.851447 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:28.852446 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:28.852774 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:28.852849 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:28.853575 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:28.853577 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:28.853577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:28.853804 (+ 227us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:28.853823 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:28.853830 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:28.853896 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:28.853958 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 23:03:28.853983 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:03:28.859395 27136 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:28.859959 27136 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:28.868523 27136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 608da53293434630b33daf5ea4c0d7e3: Generated new TSK 0
I20260810 23:03:28.868706 27136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:03:28.878118 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 27052
2026-08-10T23:03:28Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1381 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:28 dist-test-slave-1xrh krb5kdc[27161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:28 dist-test-slave-1xrh krb5kdc[27161](info): set up 2 sockets
Aug 10 23:03:28 dist-test-slave-1xrh krb5kdc[27161](info): commencing operation
krb5kdc: starting...
W20260810 23:03:29.991370 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:30.114135 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:39873
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:39873
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:30.194728 27177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:30.194979 27177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:30.195015 27177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:30.196156 27177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:30.196223 27177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:30.196245 27177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:30.196266 27177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:30.196298 27177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:30.197774 27177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:39873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:39873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:30.198071 27177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:30.198293 27177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:03:30.203116 27177 server_base.cc:1034] running on GCE node
W20260810 23:03:30.203094 27183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:30.203083 27185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:30.203089 27182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:30.203544 27177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:30.203790 27177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:30.204964 27177 hybrid_clock.cc:648] HybridClock initialized: now 1786403010204943 us; error 39 us; skew 500 ppm
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:30.207479 27177 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:30.207854 27177 webserver.cc:460] Webserver started at http://127.23.56.126:38801/ using document root <none> and password file <none>
I20260810 23:03:30.208043 27177 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:30.208084 27177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:30.208173 27177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:30.209208 27177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "8a7c2796e239432aa2191785e9e57329"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "339d76a7fdd33ce882a2c35391f439ee"
server_key_iv: "1c927953bf737f311b6630c98ab207e8"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.209507 27177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "8a7c2796e239432aa2191785e9e57329"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "339d76a7fdd33ce882a2c35391f439ee"
server_key_iv: "1c927953bf737f311b6630c98ab207e8"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.210769 27177 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:30.211503 27193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.211679 27177 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.211786 27177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "8a7c2796e239432aa2191785e9e57329"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "339d76a7fdd33ce882a2c35391f439ee"
server_key_iv: "1c927953bf737f311b6630c98ab207e8"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.211863 27177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:30.236191 27177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:30.238595 27177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:30.238765 27177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:30.243180 27177 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:39873
I20260810 23:03:30.243222 27248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:39873 every 8 connection(s)
I20260810 23:03:30.243608 27177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:30.244701 27249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:30.246958 27249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329: Bootstrap starting.
I20260810 23:03:30.247567 27249 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:30.247821 27249 log.cc:824] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:30.248520 27249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329: No bootstrap required, opened a new log
I20260810 23:03:30.249241 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 27177
I20260810 23:03:30.249358 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:30.249541 23777 external_mini_cluster.cc:1428] Setting key 19b75c8dd7f916c2a888e979bbde13c4
I20260810 23:03:30.250434 27249 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7c2796e239432aa2191785e9e57329" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39873 } }
I20260810 23:03:30.250602 27249 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:30.250645 27249 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a7c2796e239432aa2191785e9e57329, State: Initialized, Role: FOLLOWER
I20260810 23:03:30.250787 27249 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7c2796e239432aa2191785e9e57329" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39873 } }
I20260810 23:03:30.250871 27249 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:30.250908 27249 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:30.250962 27249 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:30.251745 27249 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7c2796e239432aa2191785e9e57329" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39873 } }
I20260810 23:03:30.251864 27249 leader_election.cc:302] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a7c2796e239432aa2191785e9e57329; no voters:
I20260810 23:03:30.252049 27249 leader_election.cc:288] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:30.252074 27255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:30.252184 27255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [term 1 LEADER]: Becoming Leader. State: Replica: 8a7c2796e239432aa2191785e9e57329, State: Running, Role: LEADER
I20260810 23:03:30.252317 27249 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:30.252329 27255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7c2796e239432aa2191785e9e57329" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39873 } }
I20260810 23:03:30.252777 27256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a7c2796e239432aa2191785e9e57329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7c2796e239432aa2191785e9e57329" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39873 } } }
I20260810 23:03:30.253027 27256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:30.252949 27258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a7c2796e239432aa2191785e9e57329. Latest consensus state: current_term: 1 leader_uuid: "8a7c2796e239432aa2191785e9e57329" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a7c2796e239432aa2191785e9e57329" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39873 } } }
I20260810 23:03:30.253525 27258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:30.253695 27268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:30.254298 27268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:30.255975 27268 catalog_manager.cc:1347] Generated new cluster ID: 4dc241cc1f6d44c6a4ab42f813e9db3c
I20260810 23:03:30.256032 27268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:30.262210 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:30.250095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.23.56.126:39873)
0810 23:03:30.250304 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:30.250314 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:30.250362 (+ 48us) server_negotiation.cc:407] Connection header received
0810 23:03:30.250488 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:30.250495 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:30.250550 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:30.250711 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:30.251970 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.253323 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.254519 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.255145 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.255245 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:30.258029 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:30.258033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:30.258033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:30.258046 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:30.260247 (+ 2201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.260660 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.260664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.260665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.260786 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.260985 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.260986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.260986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.261312 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:30.261527 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:30.261800 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 23:03:30.262007 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
I20260810 23:03:30.263585 27268 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:30.264271 27268 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:30.272312 27268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a7c2796e239432aa2191785e9e57329: Generated new TSK 0
I20260810 23:03:30.272481 27268 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:30.306492 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:39873
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:30.394274 27277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:30.394464 27277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:30.394488 27277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:30.395643 27277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:30.395684 27277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:30.395709 27277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:30.397141 27277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:30.397367 27277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:30.397624 27277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:30.403396 27289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:30.403411 27283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:30.403592 27277 server_base.cc:1034] running on GCE node
W20260810 23:03:30.403399 27285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:30.404115 27277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:30.404371 27277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:30.405537 27277 hybrid_clock.cc:648] HybridClock initialized: now 1786403010405522 us; error 31 us; skew 500 ppm
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:30.408025 27277 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:30.408452 27277 webserver.cc:460] Webserver started at http://127.23.56.65:41451/ using document root <none> and password file <none>
I20260810 23:03:30.408728 27277 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:30.408798 27277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:30.408957 27277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:30.410228 27277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "85ba8e07766f472ea4c3e3ed4b224dfd"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "ff5236a0722f7514bb5824ce9377d6d1"
server_key_iv: "a2d2abf694883e7d10c6799011e01a7e"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.410627 27277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "85ba8e07766f472ea4c3e3ed4b224dfd"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "ff5236a0722f7514bb5824ce9377d6d1"
server_key_iv: "a2d2abf694883e7d10c6799011e01a7e"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.412027 27277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:30.412748 27296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.412926 27277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.413018 27277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "85ba8e07766f472ea4c3e3ed4b224dfd"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "ff5236a0722f7514bb5824ce9377d6d1"
server_key_iv: "a2d2abf694883e7d10c6799011e01a7e"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.413100 27277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:30.420082 27277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:30.422298 27277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:30.422461 27277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:30.422681 27277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:30.422986 27277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:30.423038 27277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.423070 27277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:30.423096 27277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.429057 27277 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:45695
I20260810 23:03:30.429116 27418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:45695 every 8 connection(s)
I20260810 23:03:30.429401 27277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:30.431699 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 27277
I20260810 23:03:30.431818 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:30.432027 23777 external_mini_cluster.cc:1428] Setting key d5781c8a58055f3e91720ee4b95dfcfb
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:30.439707 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:30.430083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:38515 (local address 127.23.56.126:39873)
0810 23:03:30.430131 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:30.430134 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:30.430348 (+ 214us) server_negotiation.cc:407] Connection header received
0810 23:03:30.430787 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:30.430788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:30.430800 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:30.430853 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:30.431967 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.432667 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.434152 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.434431 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.434512 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:30.436598 (+ 2086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:30.436600 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:30.436600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:30.436610 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:30.438297 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.438711 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.438713 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.438713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.438797 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.439031 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.439032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.439033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.439193 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:30.439267 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:30.439517 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 23:03:30.439608 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:03:30.439726 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:30.430069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:39873 (local address 127.23.56.65:38515)
0810 23:03:30.430232 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:30.430266 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:30.430709 (+ 443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:30.430900 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:30.430903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:30.431573 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:30.431856 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:30.431863 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.432763 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.432764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:30.434058 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:30.434061 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.434533 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.434535 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:30.434672 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:30.435172 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:30.435181 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:30.436487 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:30.438379 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:30.438385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:30.438387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:30.438604 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:30.438870 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:30.438871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:30.438871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:30.438962 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:30.439257 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:30.439262 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:30.439368 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 23:03:30.439456 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:03:30.439578 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
I20260810 23:03:30.440284 27419 heartbeater.cc:347] Connected to a master server at 127.23.56.126:39873
I20260810 23:03:30.440397 27419 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:30.440569 27419 heartbeater.cc:511] Master 127.23.56.126:39873 requested a full tablet report, sending...
I20260810 23:03:30.441109 27212 ts_manager.cc:194] Registered new tserver with Master: 85ba8e07766f472ea4c3e3ed4b224dfd (127.23.56.65:45695)
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
I20260810 23:03:30.441767 27212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:38515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:30.462111 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:39873
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:30.545887 27425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:30.546113 27425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:30.546151 27425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:30.547326 27425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:30.547392 27425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:30.547436 27425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:30.548899 27425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:30.549165 27425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:30.549396 27425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:30.554543 27431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:30.554507 27425 server_base.cc:1034] running on GCE node
W20260810 23:03:30.554507 27434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:30.554564 27432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:30.555003 27425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:30.555269 27425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:30.556437 27425 hybrid_clock.cc:648] HybridClock initialized: now 1786403010556428 us; error 27 us; skew 500 ppm
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:30.558919 27425 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:30.559397 27425 webserver.cc:460] Webserver started at http://127.23.56.66:37143/ using document root <none> and password file <none>
I20260810 23:03:30.559634 27425 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:30.559703 27425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:30.559852 27425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:30.560909 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "3a2dcdd4186e4c7ea190d27b450ccdfd"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "2d622ea56e15b05b87f7f204ac25f4c4"
server_key_iv: "2c09c3f291ccf73462da603a482fcf5c"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.561256 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "3a2dcdd4186e4c7ea190d27b450ccdfd"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "2d622ea56e15b05b87f7f204ac25f4c4"
server_key_iv: "2c09c3f291ccf73462da603a482fcf5c"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.562523 27425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:03:30.563236 27441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.563409 27425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.563510 27425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "3a2dcdd4186e4c7ea190d27b450ccdfd"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "2d622ea56e15b05b87f7f204ac25f4c4"
server_key_iv: "2c09c3f291ccf73462da603a482fcf5c"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.563593 27425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:30.592011 27425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:30.594390 27425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:30.594564 27425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:30.594815 27425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:30.595189 27425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:30.595244 27425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.595284 27425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:30.595314 27425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.601315 27425 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:35367
I20260810 23:03:30.601416 27565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:35367 every 8 connection(s)
I20260810 23:03:30.601733 27425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:30.607600 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 27425
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:30.607694 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:30.607906 23777 external_mini_cluster.cc:1428] Setting key 0748048f443f9a71adddd82e860fdeee
I20260810 23:03:30.611290 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:30.602477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:39407 (local address 127.23.56.126:39873)
0810 23:03:30.602552 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:30.602555 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:30.602788 (+ 233us) server_negotiation.cc:407] Connection header received
0810 23:03:30.603316 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:30.603317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:30.603328 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:30.603383 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:30.604286 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.604784 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.605721 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.605920 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.606022 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:30.608283 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:30.608284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:30.608285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:30.608292 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:30.609694 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.610201 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.610202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.610203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.610252 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.610553 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.610554 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.610554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.610717 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:30.610791 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:30.611041 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 23:03:30.611164 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:03:30.611375 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:30.602455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:39873 (local address 127.23.56.66:39407)
0810 23:03:30.602655 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:30.602692 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:30.603227 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:30.603434 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:30.603437 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:30.604012 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:30.604207 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:30.604211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.604862 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.604863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:30.605649 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:30.605651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.606035 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.606036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:30.606174 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:30.606667 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:30.606675 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:30.608188 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:30.609776 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:30.609783 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:30.609785 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:30.610126 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:30.610319 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:30.610320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:30.610320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:30.610464 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:30.610792 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:30.610797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:30.610912 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 23:03:30.611003 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:03:30.611173 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":119,"threads_started":1}
I20260810 23:03:30.611833 27566 heartbeater.cc:347] Connected to a master server at 127.23.56.126:39873
I20260810 23:03:30.611918 27566 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:30.612120 27566 heartbeater.cc:511] Master 127.23.56.126:39873 requested a full tablet report, sending...
I20260810 23:03:30.612511 27212 ts_manager.cc:194] Registered new tserver with Master: 3a2dcdd4186e4c7ea190d27b450ccdfd (127.23.56.66:35367)
I20260810 23:03:30.613020 27212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:39407
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:30.633972 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:39873
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:30.715615 27571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:30.715858 27571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:30.715899 27571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:30.717157 27571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:30.717219 27571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:30.717264 27571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:30.718741 27571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:39873
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:30.719018 27571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:30.719285 27571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:30.724668 27576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:30.724679 27577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:30.724715 27580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:30.725003 27571 server_base.cc:1034] running on GCE node
I20260810 23:03:30.725301 27571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:30.725636 27571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:30.726883 27571 hybrid_clock.cc:648] HybridClock initialized: now 1786403010726846 us; error 42 us; skew 500 ppm
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:30.730417 27571 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:30.730968 27571 webserver.cc:460] Webserver started at http://127.23.56.67:42347/ using document root <none> and password file <none>
I20260810 23:03:30.731316 27571 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:30.731396 27571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:30.731588 27571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:30.732802 27571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "84028e3519174bb3b25a1a6419c3b1e1"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "d7efcacb6d20e8f46db599b9f77ecb84"
server_key_iv: "d8675bd2ba9b60fcb2a79166f3b1f2db"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.733229 27571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "84028e3519174bb3b25a1a6419c3b1e1"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "d7efcacb6d20e8f46db599b9f77ecb84"
server_key_iv: "d8675bd2ba9b60fcb2a79166f3b1f2db"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.734702 27571 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 23:03:30.735600 27587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.735771 27571 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.735880 27571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "84028e3519174bb3b25a1a6419c3b1e1"
format_stamp: "Formatted at 2026-08-10 23:03:30 on dist-test-slave-1xrh"
server_key: "d7efcacb6d20e8f46db599b9f77ecb84"
server_key_iv: "d8675bd2ba9b60fcb2a79166f3b1f2db"
server_key_version: "encryptionkey@0"
I20260810 23:03:30.735991 27571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:30.769398 27571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:30.771497 27571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:30.771672 27571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:30.771942 27571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:30.772377 27571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:30.772437 27571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.772478 27571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:30.772512 27571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:30.778674 27571 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:46733
I20260810 23:03:30.778717 27712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:46733 every 8 connection(s)
I20260810 23:03:30.779116 27571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:30.779510 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 27571
I20260810 23:03:30.779598 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:30.779765 23777 external_mini_cluster.cc:1428] Setting key fdc5e0e1470ac2de479fb393dd54e1ae
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:30.788266 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:30.780117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:52369 (local address 127.23.56.126:39873)
0810 23:03:30.780170 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:30.780173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:30.780395 (+ 222us) server_negotiation.cc:407] Connection header received
0810 23:03:30.781069 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:30.781070 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:30.781081 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:30.781126 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:30.782010 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.782497 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.783541 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.783724 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.783787 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:30.785902 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:30.785904 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:30.785904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:30.785911 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:30.787112 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.787516 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.787517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.787517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.787568 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.787769 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.787769 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.787769 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.787866 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:30.787915 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:30.788141 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 23:03:30.788201 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:03:30.788478 27715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:30.780116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:39873 (local address 127.23.56.67:52369)
0810 23:03:30.780276 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:30.780319 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:30.780953 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:30.781183 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:30.781187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:30.781742 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:30.781932 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:30.781936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.782575 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.782576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:30.783468 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:30.783471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.783804 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.783805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:30.783902 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:30.784446 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:30.784454 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:30.785818 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:30.787231 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:30.787237 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:30.787238 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:30.787453 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:30.787625 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:30.787626 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:30.787626 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:30.787718 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:30.787918 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:30.787922 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:30.788034 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 23:03:30.788115 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:03:30.788230 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
I20260810 23:03:30.788933 27713 heartbeater.cc:347] Connected to a master server at 127.23.56.126:39873
I20260810 23:03:30.789028 27713 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:30.789179 27713 heartbeater.cc:511] Master 127.23.56.126:39873 requested a full tablet report, sending...
I20260810 23:03:30.789537 27212 ts_manager.cc:194] Registered new tserver with Master: 84028e3519174bb3b25a1a6419c3b1e1 (127.23.56.67:46733)
I20260810 23:03:30.789862 27212 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:52369
I20260810 23:03:30.791658 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27161](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403010, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:30.814628 27254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:30.807742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.23.56.126:39873)
0810 23:03:30.807826 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:30.807828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:30.807930 (+ 102us) server_negotiation.cc:407] Connection header received
0810 23:03:30.807982 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:30.807983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:30.807995 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:30.808046 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:30.808641 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.809182 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.809873 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:30.810055 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:30.810105 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:30.812384 (+ 2279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:30.812385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:30.812386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:30.812393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:30.813589 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.813912 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.813912 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.813913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.813962 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:30.814127 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:30.814128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:30.814128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:30.814270 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:30.814344 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:30.814444 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 23:03:30.814516 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 23:03:30.814889 27212 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:42836
I20260810 23:03:30.815459 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 27277
I20260810 23:03:30.822152 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 27425
I20260810 23:03:30.834563 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 27571
I20260810 23:03:30.841322 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 27177
2026-08-10T23:03:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1965 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27727](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27727](info): set up 2 sockets
Aug 10 23:03:30 dist-test-slave-1xrh krb5kdc[27727](info): commencing operation
krb5kdc: starting...
W20260810 23:03:31.941057 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:32.066222 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:42945
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:42945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:32.149385 27741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:32.149647 27741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:32.149688 27741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:32.150889 27741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:32.150965 27741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:32.150991 27741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:32.151021 27741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:32.151052 27741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:32.152721 27741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:42945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:32.153048 27741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:32.153302 27741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:03:32.158540 27741 server_base.cc:1034] running on GCE node
W20260810 23:03:32.158532 27749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:32.158473 27747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:32.158473 27746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:32.158982 27741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:32.159314 27741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:32.160493 27741 hybrid_clock.cc:648] HybridClock initialized: now 1786403012160472 us; error 30 us; skew 500 ppm
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:32.163044 27741 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:32.163656 27741 webserver.cc:460] Webserver started at http://127.23.56.126:45739/ using document root <none> and password file <none>
I20260810 23:03:32.164047 27741 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:32.164121 27741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:32.164268 27741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:32.165817 27741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "d261a05363124acf94a1eeffd6cdfcf9"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "96d61b934fa31789c3578f1101944fbb"
server_key_iv: "1b698627a320bd44757cca32812ca76b"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.166137 27741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "d261a05363124acf94a1eeffd6cdfcf9"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "96d61b934fa31789c3578f1101944fbb"
server_key_iv: "1b698627a320bd44757cca32812ca76b"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.167886 27741 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
I20260810 23:03:32.168738 27757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.168917 27741 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:32.169026 27741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "d261a05363124acf94a1eeffd6cdfcf9"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "96d61b934fa31789c3578f1101944fbb"
server_key_iv: "1b698627a320bd44757cca32812ca76b"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.169111 27741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:32.186123 27741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:32.188524 27741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:32.188702 27741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:32.192996 27741 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:42945
I20260810 23:03:32.193039 27812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:42945 every 8 connection(s)
I20260810 23:03:32.193600 27741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:32.194298 27814 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:32.196653 27814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9: Bootstrap starting.
I20260810 23:03:32.197279 27814 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:32.197598 27814 log.cc:824] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:32.198324 27814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9: No bootstrap required, opened a new log
I20260810 23:03:32.199985 27814 raft_consensus.cc:348] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261a05363124acf94a1eeffd6cdfcf9" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42945 } }
I20260810 23:03:32.200124 27814 raft_consensus.cc:374] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:32.200146 27814 raft_consensus.cc:729] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d261a05363124acf94a1eeffd6cdfcf9, State: Initialized, Role: FOLLOWER
I20260810 23:03:32.200320 27814 consensus_queue.cc:260] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261a05363124acf94a1eeffd6cdfcf9" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42945 } }
I20260810 23:03:32.200393 27814 raft_consensus.cc:388] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:32.200414 27814 raft_consensus.cc:482] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:32.200479 27814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:32.201385 27814 raft_consensus.cc:504] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261a05363124acf94a1eeffd6cdfcf9" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42945 } }
I20260810 23:03:32.201473 27814 leader_election.cc:302] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d261a05363124acf94a1eeffd6cdfcf9; no voters:
I20260810 23:03:32.201637 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 27741
I20260810 23:03:32.201692 27814 leader_election.cc:288] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:32.201756 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:32.201756 27818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:32.201937 27818 raft_consensus.cc:686] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [term 1 LEADER]: Becoming Leader. State: Replica: d261a05363124acf94a1eeffd6cdfcf9, State: Running, Role: LEADER
I20260810 23:03:32.201965 27814 sys_catalog.cc:564] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:32.202028 23777 external_mini_cluster.cc:1428] Setting key bcfc31b965893da3e97da53b2bbe6591
I20260810 23:03:32.202097 27818 consensus_queue.cc:237] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261a05363124acf94a1eeffd6cdfcf9" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42945 } }
I20260810 23:03:32.202778 27819 sys_catalog.cc:455] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d261a05363124acf94a1eeffd6cdfcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261a05363124acf94a1eeffd6cdfcf9" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42945 } } }
I20260810 23:03:32.202869 27819 sys_catalog.cc:458] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:32.203038 27820 sys_catalog.cc:455] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d261a05363124acf94a1eeffd6cdfcf9. Latest consensus state: current_term: 1 leader_uuid: "d261a05363124acf94a1eeffd6cdfcf9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d261a05363124acf94a1eeffd6cdfcf9" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42945 } } }
I20260810 23:03:32.203099 27820 sys_catalog.cc:458] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9 [sys.catalog]: This master's current role is: LEADER
W20260810 23:03:32.203933 27838 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:03:32.204010 27838 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:03:32.204109 27829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:32.204285 27829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:32.205703 27829 catalog_manager.cc:1347] Generated new cluster ID: ce263ec1b88c4923a2957dd448750849
I20260810 23:03:32.205754 27829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:32.214251 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:32.202534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.23.56.126:42945)
0810 23:03:32.202709 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:32.202720 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:32.202966 (+ 246us) server_negotiation.cc:407] Connection header received
0810 23:03:32.203211 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:32.203219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:32.203287 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:32.203404 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:32.204417 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.205811 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.206850 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.207541 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.207666 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:32.210545 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:32.210550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:32.210550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:32.210564 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:32.212570 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.212990 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.212994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.212994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.213097 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.213321 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.213322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.213322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.213629 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:32.213776 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:32.213905 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 23:03:32.214041 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":90,"threads_started":1}
I20260810 23:03:32.232399 27829 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:32.233246 27829 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:32.239806 27829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d261a05363124acf94a1eeffd6cdfcf9: Generated new TSK 0
I20260810 23:03:32.240037 27829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:32.274627 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:42945
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:32.360340 27843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:32.360564 27843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:32.360589 27843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:32.361784 27843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:32.361831 27843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:32.361860 27843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:32.363476 27843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:42945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:32.363715 27843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:32.363986 27843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:32.369240 27852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:32.369403 27843 server_base.cc:1034] running on GCE node
W20260810 23:03:32.369270 27849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:32.369254 27848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:32.369801 27843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:32.370066 27843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:32.371237 27843 hybrid_clock.cc:648] HybridClock initialized: now 1786403012371226 us; error 47 us; skew 500 ppm
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:32.374014 27843 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:32.374502 27843 webserver.cc:460] Webserver started at http://127.23.56.65:43281/ using document root <none> and password file <none>
I20260810 23:03:32.374747 27843 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:32.374825 27843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:32.374992 27843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:32.376220 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "11ccc2b1a2c84a5a985be2ba0706c228"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "73c5d72d893e96f4dd192a7a21267473"
server_key_iv: "03837312921725664fcc7ead8b0e2a2b"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.376645 27843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "11ccc2b1a2c84a5a985be2ba0706c228"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "73c5d72d893e96f4dd192a7a21267473"
server_key_iv: "03837312921725664fcc7ead8b0e2a2b"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.378054 27843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:32.378769 27859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.378957 27843 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:32.379063 27843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "11ccc2b1a2c84a5a985be2ba0706c228"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "73c5d72d893e96f4dd192a7a21267473"
server_key_iv: "03837312921725664fcc7ead8b0e2a2b"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.379161 27843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:32.404368 27843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:32.406273 27843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:32.406453 27843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:32.406718 27843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:32.407069 27843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:32.407126 27843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.407239 27843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:32.407269 27843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.413125 27843 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:42651
I20260810 23:03:32.413179 27982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:42651 every 8 connection(s)
I20260810 23:03:32.413496 27843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:32.420815 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 27843
I20260810 23:03:32.420930 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:32.421129 23777 external_mini_cluster.cc:1428] Setting key 59effd07a314bcdef73300500b0c5e59
I20260810 23:03:32.422827 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:32.414235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:47003 (local address 127.23.56.126:42945)
0810 23:03:32.414283 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:32.414285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:32.414488 (+ 203us) server_negotiation.cc:407] Connection header received
0810 23:03:32.415426 (+ 938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:32.415427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:32.415441 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:32.415500 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:32.416082 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.416583 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.417577 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.417763 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.417814 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:32.419880 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:32.419882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:32.419882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:32.419888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:32.421353 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.421749 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.421750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.421750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.421816 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.422112 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.422114 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.422114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.422243 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:32.422350 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:32.422641 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 23:03:32.422733 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:03:32.423041 27986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:32.414186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:42945 (local address 127.23.56.65:47003)
0810 23:03:32.414348 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:32.414391 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:32.415328 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:32.415554 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:32.415557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:32.415808 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:32.416007 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:32.416011 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.416662 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.416663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:32.417494 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:32.417496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.417838 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.417839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:32.417972 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:32.418409 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:32.418418 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:32.419790 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:32.421430 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:32.421438 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:32.421441 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:32.421674 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:32.421910 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:32.421911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:32.421911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:32.422029 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:32.422343 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:32.422350 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:32.422492 (+ 142us) client_negotiation.cc:769] Sending connection context
0810 23:03:32.422609 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 23:03:32.422781 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260810 23:03:32.423602 27983 heartbeater.cc:347] Connected to a master server at 127.23.56.126:42945
I20260810 23:03:32.423759 27983 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:32.423982 27983 heartbeater.cc:511] Master 127.23.56.126:42945 requested a full tablet report, sending...
I20260810 23:03:32.424541 27776 ts_manager.cc:194] Registered new tserver with Master: 11ccc2b1a2c84a5a985be2ba0706c228 (127.23.56.65:42651)
I20260810 23:03:32.425149 27776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:47003
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:32.454205 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:42945
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:32.564527 27989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:32.564888 27989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:32.564920 27989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:32.566416 27989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:32.566493 27989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:32.566530 27989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:32.569125 27989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:42945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:32.569563 27989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:32.569962 27989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:32.576457 27995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:32.576534 27998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:32.576824 27989 server_base.cc:1034] running on GCE node
W20260810 23:03:32.577011 27996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:32.577353 27989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:32.577662 27989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:32.578922 27989 hybrid_clock.cc:648] HybridClock initialized: now 1786403012578911 us; error 35 us; skew 500 ppm
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:32.582204 27989 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:32.582723 27989 webserver.cc:460] Webserver started at http://127.23.56.66:38193/ using document root <none> and password file <none>
I20260810 23:03:32.583010 27989 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:32.583086 27989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:32.583276 27989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:32.584403 27989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "8a273e2f977745b39249f7266c0e7ec9"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "e4414742d3ce54808daede25ceb300e0"
server_key_iv: "46c3d1bc7a24201b1ac373c2c8a1214c"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.584770 27989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "8a273e2f977745b39249f7266c0e7ec9"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "e4414742d3ce54808daede25ceb300e0"
server_key_iv: "46c3d1bc7a24201b1ac373c2c8a1214c"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.586163 27989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:03:32.587001 28006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.587220 27989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:32.587292 27989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "8a273e2f977745b39249f7266c0e7ec9"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "e4414742d3ce54808daede25ceb300e0"
server_key_iv: "46c3d1bc7a24201b1ac373c2c8a1214c"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.587387 27989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:32.613034 27989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:32.615048 27989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:32.615250 27989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:32.615521 27989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:32.615860 27989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:32.615895 27989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.615918 27989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:32.615969 27989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.621646 27989 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:43729
I20260810 23:03:32.621734 28126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:43729 every 8 connection(s)
I20260810 23:03:32.622107 27989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:32.631363 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:32.622934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:51975 (local address 127.23.56.126:42945)
0810 23:03:32.623015 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:32.623017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:32.623202 (+ 185us) server_negotiation.cc:407] Connection header received
0810 23:03:32.624142 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:32.624143 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:32.624154 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:32.624209 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:32.624783 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.625283 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.626230 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.626439 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.626500 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:32.628642 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:32.628644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:32.628644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:32.628651 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:32.630176 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.630516 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.630517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.630517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.630569 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.630767 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.630768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.630768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.630892 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:32.630943 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:32.631233 (+ 290us) server_negotiation.cc:299] Negotiation successful
0810 23:03:32.631298 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:03:32.631649 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:32.622885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:42945 (local address 127.23.56.66:51975)
0810 23:03:32.623062 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:32.623106 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:32.624051 (+ 945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:32.624274 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:32.624277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:32.624524 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:32.624715 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:32.624719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.625362 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.625363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:32.626146 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:32.626148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.626520 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.626521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:32.626618 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:32.627231 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:32.627239 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:32.628547 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:32.630244 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:32.630249 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:32.630251 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:32.630446 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:32.630625 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:32.630625 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:32.630626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:32.630714 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:32.630947 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:32.630951 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:32.631068 (+ 117us) client_negotiation.cc:769] Sending connection context
0810 23:03:32.631217 (+ 149us) client_negotiation.cc:240] Negotiation successful
0810 23:03:32.631336 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260810 23:03:32.631637 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 27989
I20260810 23:03:32.631748 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:32.631929 23777 external_mini_cluster.cc:1428] Setting key ce6b6d68f9e47eaaa784f40fe4992aca
I20260810 23:03:32.632280 28127 heartbeater.cc:347] Connected to a master server at 127.23.56.126:42945
I20260810 23:03:32.632359 28127 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:32.632567 28127 heartbeater.cc:511] Master 127.23.56.126:42945 requested a full tablet report, sending...
I20260810 23:03:32.632987 27776 ts_manager.cc:194] Registered new tserver with Master: 8a273e2f977745b39249f7266c0e7ec9 (127.23.56.66:43729)
I20260810 23:03:32.633484 27776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:51975
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:32.658630 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:42945
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:34821
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:32.741473 28132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:32.741677 28132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:32.741700 28132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:32.742807 28132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:32.742849 28132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:32.742875 28132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:32.744467 28132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:34821
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:42945
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:32.744661 28132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:32.744848 28132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:32.749789 28137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:32.749928 28132 server_base.cc:1034] running on GCE node
W20260810 23:03:32.749789 28140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:32.749838 28138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:32.750298 28132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:32.750577 28132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:32.751735 28132 hybrid_clock.cc:648] HybridClock initialized: now 1786403012751720 us; error 26 us; skew 500 ppm
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:32.754433 28132 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:32.754869 28132 webserver.cc:460] Webserver started at http://127.23.56.67:36667/ using document root <none> and password file <none>
I20260810 23:03:32.755110 28132 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:32.755218 28132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:32.755376 28132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:32.756434 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "4f3ff7e18fa74781beb6ac6968093ea0"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "968a70881c5cd05438ed6d3b9e4c002b"
server_key_iv: "9836b12551da7b3eb1c1682a7535eeb3"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.756816 28132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "4f3ff7e18fa74781beb6ac6968093ea0"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "968a70881c5cd05438ed6d3b9e4c002b"
server_key_iv: "9836b12551da7b3eb1c1682a7535eeb3"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.758236 28132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:32.759057 28147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.759253 28132 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:32.759366 28132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "4f3ff7e18fa74781beb6ac6968093ea0"
format_stamp: "Formatted at 2026-08-10 23:03:32 on dist-test-slave-1xrh"
server_key: "968a70881c5cd05438ed6d3b9e4c002b"
server_key_iv: "9836b12551da7b3eb1c1682a7535eeb3"
server_key_version: "encryptionkey@0"
I20260810 23:03:32.759441 28132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:32.780229 28132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:32.782781 28132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:32.782977 28132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:32.783299 28132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:32.783633 28132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:32.783687 28132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.783731 28132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:32.783758 28132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:32.789361 28132 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:38001
I20260810 23:03:32.789460 28268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:38001 every 8 connection(s)
I20260810 23:03:32.789768 28132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:32.793889 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 28132
I20260810 23:03:32.794031 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:32.794211 23777 external_mini_cluster.cc:1428] Setting key bca05aa23676fa7e12c74711b4662a01
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:32.798743 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:32.790602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:41921 (local address 127.23.56.126:42945)
0810 23:03:32.790685 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:32.790687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:32.790836 (+ 149us) server_negotiation.cc:407] Connection header received
0810 23:03:32.791854 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:32.791855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:32.791865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:32.791918 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:32.792503 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.792980 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.793955 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.794254 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.794327 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:32.796566 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:32.796568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:32.796568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:32.796574 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:32.797674 (+ 1100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.798027 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.798027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.798028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.798076 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.798242 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.798243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.798243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.798336 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:32.798385 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:32.798620 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 23:03:32.798681 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:03:32.798933 28271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:32.790556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:42945 (local address 127.23.56.67:41921)
0810 23:03:32.790712 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:32.790748 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:32.791762 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:32.791973 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:32.791977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:32.792233 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:32.792428 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:32.792432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.793070 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.793071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:32.793865 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:32.793868 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.794359 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.794361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:32.794524 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:32.795087 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:32.795098 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:32.796477 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:32.797743 (+ 1266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:32.797749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:32.797751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:32.797961 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:32.798115 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:32.798115 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:32.798116 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:32.798195 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:32.798392 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:32.798397 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:32.798510 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 23:03:32.798592 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:03:32.798704 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260810 23:03:32.799427 28269 heartbeater.cc:347] Connected to a master server at 127.23.56.126:42945
I20260810 23:03:32.799540 28269 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:32.799715 28269 heartbeater.cc:511] Master 127.23.56.126:42945 requested a full tablet report, sending...
I20260810 23:03:32.800096 27776 ts_manager.cc:194] Registered new tserver with Master: 4f3ff7e18fa74781beb6ac6968093ea0 (127.23.56.67:38001)
I20260810 23:03:32.800421 27776 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:41921
I20260810 23:03:32.806003 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:32.829380 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:32.821963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56628 (local address 127.23.56.65:42651)
0810 23:03:32.822137 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:32.822147 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:32.822158 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:03:32.822185 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:32.822187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:32.822202 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:32.822271 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:32.822808 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.823597 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.824329 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.824635 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.824688 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:32.826776 (+ 2088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:32.826784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:32.826784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:32.826791 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:32.828401 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.828689 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.828692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.828692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.828744 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.828884 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.828884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.828884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.829013 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:32.829108 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:32.829178 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 23:03:32.829240 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[27727](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403012, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:32.836938 27824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:32.830676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46944 (local address 127.23.56.126:42945)
0810 23:03:32.830748 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:32.830750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:32.830760 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:32.830784 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:32.830785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:32.830794 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:32.830839 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:32.831319 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.831782 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.832384 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:32.832565 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:32.832613 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:32.834566 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:32.834568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:32.834568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:32.834576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:32.835793 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.836138 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.836140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.836140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.836197 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:32.836398 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:32.836399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:32.836400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:32.836554 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:32.836656 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:32.836765 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 23:03:32.836838 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 23:03:32.837169 27776 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46944
I20260810 23:03:32.837527 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 27843
I20260810 23:03:32.851928 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 27989
I20260810 23:03:32.858928 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 28132
I20260810 23:03:32.864610 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 27741
2026-08-10T23:03:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2021 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[28281](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[28281](info): set up 2 sockets
Aug 10 23:03:32 dist-test-slave-1xrh krb5kdc[28281](info): commencing operation
krb5kdc: starting...
W20260810 23:03:33.971351 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:34 dist-test-slave-1xrh krb5kdc[28281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:34.094707 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33931
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:34.175603 28295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:34.175827 28295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:34.175864 28295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:34.176980 28295 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:34.177040 28295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:34.177067 28295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:34.177098 28295 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:34.177125 28295 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:34.178581 28295 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:34.178843 28295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:34.179059 28295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:34.183770 28303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:34.183756 28300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:34.183974 28295 server_base.cc:1034] running on GCE node
W20260810 23:03:34.183841 28301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:34.184244 28295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:34.184466 28295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:34.185604 28295 hybrid_clock.cc:648] HybridClock initialized: now 1786403014185585 us; error 36 us; skew 500 ppm
Aug 10 23:03:34 dist-test-slave-1xrh krb5kdc[28281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:34.193254 28295 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:34.193636 28295 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:34.193686 28295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:34.193851 28295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:34.194869 28295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "74d8d5f097c14241b02c8846fe96463f"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "d9ce09b8407fbbda6fc663a3e2e3fb30"
server_key_iv: "75f4444094df6dec7e86a42baebf745f"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.195261 28295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "74d8d5f097c14241b02c8846fe96463f"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "d9ce09b8407fbbda6fc663a3e2e3fb30"
server_key_iv: "75f4444094df6dec7e86a42baebf745f"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.196521 28295 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:34.197218 28309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.197379 28295 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:34.197494 28295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "74d8d5f097c14241b02c8846fe96463f"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "d9ce09b8407fbbda6fc663a3e2e3fb30"
server_key_iv: "75f4444094df6dec7e86a42baebf745f"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.197567 28295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:34.211665 28295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:34.213311 28295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:34.213474 28295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:34.217531 28295 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33931
I20260810 23:03:34.217559 28372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33931 every 8 connection(s)
I20260810 23:03:34.217824 28295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:34.218802 28373 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:34.219383 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 28295
I20260810 23:03:34.219465 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:34.219650 23777 external_mini_cluster.cc:1428] Setting key f3e423926a5591f045ec4989c8c9d11a
I20260810 23:03:34.221756 28373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f: Bootstrap starting.
I20260810 23:03:34.222369 28373 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:34.222643 28373 log.cc:824] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:34.223419 28373 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f: No bootstrap required, opened a new log
I20260810 23:03:34.225734 28373 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d8d5f097c14241b02c8846fe96463f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33931 } }
I20260810 23:03:34.225907 28373 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:34.225943 28373 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74d8d5f097c14241b02c8846fe96463f, State: Initialized, Role: FOLLOWER
I20260810 23:03:34.226080 28373 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d8d5f097c14241b02c8846fe96463f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33931 } }
I20260810 23:03:34.226163 28373 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:34.226193 28373 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:34.226240 28373 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [term 0 FOLLOWER]: Advancing to term 1
Aug 10 23:03:34 dist-test-slave-1xrh krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:34.226835 28373 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d8d5f097c14241b02c8846fe96463f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33931 } }
I20260810 23:03:34.226946 28373 leader_election.cc:302] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74d8d5f097c14241b02c8846fe96463f; no voters:
I20260810 23:03:34.227104 28373 leader_election.cc:288] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:34.227312 28380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:34.227418 28373 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:34.227497 28380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [term 1 LEADER]: Becoming Leader. State: Replica: 74d8d5f097c14241b02c8846fe96463f, State: Running, Role: LEADER
I20260810 23:03:34.227651 28380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d8d5f097c14241b02c8846fe96463f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33931 } }
I20260810 23:03:34.228055 28380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74d8d5f097c14241b02c8846fe96463f. Latest consensus state: current_term: 1 leader_uuid: "74d8d5f097c14241b02c8846fe96463f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d8d5f097c14241b02c8846fe96463f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33931 } } }
I20260810 23:03:34.228307 28380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:34.228240 28379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74d8d5f097c14241b02c8846fe96463f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74d8d5f097c14241b02c8846fe96463f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33931 } } }
I20260810 23:03:34.228646 28379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:34.228967 28389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:34.229475 28389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:34.230813 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.220153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60492 (local address 127.23.56.126:33931)
0810 23:03:34.220325 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.220332 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.220355 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:03:34.220738 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.220744 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.220786 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.220895 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:34.221715 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.222658 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.223615 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.224277 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.224394 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.226759 (+ 2365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:34.226763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:34.226763 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:34.226775 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:34.228823 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.229264 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.229268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.229268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.229374 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.229594 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.229595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.229595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.229885 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:34.230014 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.230550 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.230666 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":89,"threads_started":1}
I20260810 23:03:34.231213 28389 catalog_manager.cc:1347] Generated new cluster ID: 3e691917de33433097ef183a00a78f7c
I20260810 23:03:34.231266 28389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:34.238044 28389 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:34.238580 28389 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:34.246793 28389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f: Generated new TSK 0
I20260810 23:03:34.246963 28389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:34.277832 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33931
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:34.366933 28401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:34.367184 28401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:34.367224 28401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:34.368382 28401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:34.368445 28401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:34.368494 28401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:34.370035 28401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:34.370289 28401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:34.370519 28401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:34.375458 28410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:34.375485 28407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:34.375453 28408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:34.375677 28401 server_base.cc:1034] running on GCE node
I20260810 23:03:34.375934 28401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:34.376184 28401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:34.377350 28401 hybrid_clock.cc:648] HybridClock initialized: now 1786403014377344 us; error 35 us; skew 500 ppm
Aug 10 23:03:34 dist-test-slave-1xrh krb5kdc[28281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:34.380143 28401 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:34.380496 28401 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:34.380581 28401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:34.380740 28401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:34.381883 28401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "ccadff7636fc4731ad02e65f9a0f000f"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "7af50e170c39e44e9b078b6cc89ded47"
server_key_iv: "f44dd2525f85a04abc528133700687c7"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.382253 28401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "ccadff7636fc4731ad02e65f9a0f000f"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "7af50e170c39e44e9b078b6cc89ded47"
server_key_iv: "f44dd2525f85a04abc528133700687c7"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.383625 28401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:34.384485 28416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.384703 28401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:34.384811 28401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "ccadff7636fc4731ad02e65f9a0f000f"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "7af50e170c39e44e9b078b6cc89ded47"
server_key_iv: "f44dd2525f85a04abc528133700687c7"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.384924 28401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:34.392428 28401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:34.394172 28401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:34.394320 28401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:34.394554 28401 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:34.394862 28401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:34.394896 28401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.394917 28401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:34.394966 28401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.400852 28401 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:41255
I20260810 23:03:34.400944 28541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:41255 every 8 connection(s)
I20260810 23:03:34.401190 28401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:34.403232 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 28401
I20260810 23:03:34.403360 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:34.403525 23777 external_mini_cluster.cc:1428] Setting key 50df243d2613ce64b12da146e2b7c76d
Aug 10 23:03:34 dist-test-slave-1xrh krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:34.411116 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.401853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:54969 (local address 127.23.56.126:33931)
0810 23:03:34.401932 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.401935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.402094 (+ 159us) server_negotiation.cc:407] Connection header received
0810 23:03:34.402562 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.402563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.402573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.402621 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:34.403652 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.404324 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.405329 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.405553 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.405641 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.408088 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:34.408089 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:34.408089 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:34.408096 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:34.409708 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.410108 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.410109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.410110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.410178 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.410413 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.410414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.410415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.410583 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:34.410660 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.410899 (+ 239us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.410997 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:03:34.411162 28544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.401816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33931 (local address 127.23.56.65:54969)
0810 23:03:34.401978 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.402013 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:34.402480 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:34.402678 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:34.402681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:34.402922 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:34.403536 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.403543 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.404417 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.404418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.405223 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.405225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.405652 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.405653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.405786 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.406258 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:34.406266 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:34.407974 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:34.409797 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:34.409804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:34.409806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:34.410027 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:34.410252 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:34.410253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:34.410253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:34.410345 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:34.410653 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:34.410657 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:34.410763 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 23:03:34.410844 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:03:34.410956 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260810 23:03:34.411687 28542 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33931
I20260810 23:03:34.411800 28542 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:34.411954 28542 heartbeater.cc:511] Master 127.23.56.126:33931 requested a full tablet report, sending...
I20260810 23:03:34.412472 28326 ts_manager.cc:194] Registered new tserver with Master: ccadff7636fc4731ad02e65f9a0f000f (127.23.56.65:41255)
I20260810 23:03:34.413087 28326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:54969
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:34.430951 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33931
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:34.512985 28547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:34.513219 28547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:34.513258 28547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:34.514405 28547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:34.514467 28547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:34.514510 28547 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:34.516075 28547 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:34.516337 28547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:34.516562 28547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:34.521456 28553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:34.521459 28554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:34.521485 28556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:34.521685 28547 server_base.cc:1034] running on GCE node
I20260810 23:03:34.521940 28547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:34.522172 28547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:34.523332 28547 hybrid_clock.cc:648] HybridClock initialized: now 1786403014523320 us; error 44 us; skew 500 ppm
Aug 10 23:03:34 dist-test-slave-1xrh krb5kdc[28281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:34.525719 28547 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:34.526096 28547 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:34.526170 28547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:34.526322 28547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:34.527381 28547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "416506d026894fbfaa7f6062d01b56e9"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "dc8147bb154767069e4dab3d8c5d3a8d"
server_key_iv: "54bcd480cd545586051fc13562df56f9"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.527770 28547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "416506d026894fbfaa7f6062d01b56e9"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "dc8147bb154767069e4dab3d8c5d3a8d"
server_key_iv: "54bcd480cd545586051fc13562df56f9"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.529066 28547 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:34.529734 28562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.529901 28547 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:34.530009 28547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "416506d026894fbfaa7f6062d01b56e9"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "dc8147bb154767069e4dab3d8c5d3a8d"
server_key_iv: "54bcd480cd545586051fc13562df56f9"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.530088 28547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:34.562188 28547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:34.564255 28547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:34.564426 28547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:34.564666 28547 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:34.564994 28547 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:34.565061 28547 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.565100 28547 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:34.565127 28547 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.571233 28547 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:34763
I20260810 23:03:34.571301 28692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:34763 every 8 connection(s)
I20260810 23:03:34.571547 28547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:34.576534 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 28547
I20260810 23:03:34.576645 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:34.576844 23777 external_mini_cluster.cc:1428] Setting key f6ab6d913f6d4d2cb4678117a67710a7
Aug 10 23:03:34 dist-test-slave-1xrh krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:34.580747 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.572226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:42425 (local address 127.23.56.126:33931)
0810 23:03:34.572300 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.572303 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.572536 (+ 233us) server_negotiation.cc:407] Connection header received
0810 23:03:34.572995 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.572996 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.573007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.573058 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:34.573929 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.574410 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.575496 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.575710 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.575762 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.578016 (+ 2254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:34.578017 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:34.578018 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:34.578026 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:34.579361 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.579780 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.579782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.579782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.579843 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.580103 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.580104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.580104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.580217 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:34.580295 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.580572 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.580656 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:03:34.580929 28695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.572218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33931 (local address 127.23.56.66:42425)
0810 23:03:34.572412 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.572449 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:34.572910 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:34.573110 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:34.573114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:34.573361 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:34.573851 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.573856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.574509 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.574510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.575400 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.575402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.575781 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.575782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.575872 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.576352 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:34.576362 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:34.577918 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:34.579453 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:34.579460 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:34.579463 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:34.579687 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:34.579912 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:34.579913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:34.579913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:34.580027 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:34.580287 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:34.580294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:34.580431 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 23:03:34.580549 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:03:34.580707 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260810 23:03:34.581501 28693 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33931
I20260810 23:03:34.581612 28693 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:34.581794 28693 heartbeater.cc:511] Master 127.23.56.126:33931 requested a full tablet report, sending...
I20260810 23:03:34.582204 28326 ts_manager.cc:194] Registered new tserver with Master: 416506d026894fbfaa7f6062d01b56e9 (127.23.56.66:34763)
I20260810 23:03:34.582567 28326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:42425
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:34.603767 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33931
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:46487
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:34.685240 28698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:34.685438 28698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:34.685461 28698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:34.686653 28698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:34.686697 28698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:34.686748 28698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:34.688413 28698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:46487
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:34.688661 28698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:34.688905 28698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:34.693819 28704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:34.693851 28698 server_base.cc:1034] running on GCE node
W20260810 23:03:34.693828 28703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:34.693806 28706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:34.694192 28698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:34.694430 28698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:34.695587 28698 hybrid_clock.cc:648] HybridClock initialized: now 1786403014695574 us; error 23 us; skew 500 ppm
Aug 10 23:03:34 dist-test-slave-1xrh krb5kdc[28281](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:34.698165 28698 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:34.698451 28698 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:34.698494 28698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:34.698581 28698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:34.699611 28698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "07b0cccde8ed4698aa2003a58cd534a9"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "97bf27738b9bd88a2366099f78f48a2b"
server_key_iv: "b8d7eae12e7ff6bda5bc861d4c55d5aa"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.699978 28698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "07b0cccde8ed4698aa2003a58cd534a9"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "97bf27738b9bd88a2366099f78f48a2b"
server_key_iv: "b8d7eae12e7ff6bda5bc861d4c55d5aa"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.701303 28698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:34.701929 28714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.702102 28698 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:34.702169 28698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "07b0cccde8ed4698aa2003a58cd534a9"
format_stamp: "Formatted at 2026-08-10 23:03:34 on dist-test-slave-1xrh"
server_key: "97bf27738b9bd88a2366099f78f48a2b"
server_key_iv: "b8d7eae12e7ff6bda5bc861d4c55d5aa"
server_key_version: "encryptionkey@0"
I20260810 23:03:34.702260 28698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:34.722858 28698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:34.725194 28698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:34.725355 28698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:34.725579 28698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:34.725893 28698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:34.725926 28698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.725968 28698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:34.726002 28698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:34.732396 28698 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:46231
I20260810 23:03:34.732429 28839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:46231 every 8 connection(s)
I20260810 23:03:34.732743 28698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:34.738273 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 28698
I20260810 23:03:34.738385 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
Aug 10 23:03:34 dist-test-slave-1xrh krb5kdc[28281](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403014, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:34.738569 23777 external_mini_cluster.cc:1428] Setting key bd950d59a1b1f2a0094c23b552dea001
I20260810 23:03:34.741907 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.733436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:38483 (local address 127.23.56.126:33931)
0810 23:03:34.733481 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.733483 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.733730 (+ 247us) server_negotiation.cc:407] Connection header received
0810 23:03:34.734176 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.734177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.734186 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.734241 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:34.735089 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.735616 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.736567 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.736739 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.736787 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.739035 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:34.739037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:34.739037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:34.739043 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:34.740546 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.740957 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.740958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.740959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.741022 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.741276 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.741277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.741278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.741398 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:34.741472 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.741737 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.741816 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:03:34.742092 28842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.733419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33931 (local address 127.23.56.67:38483)
0810 23:03:34.733567 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.733647 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:34.734097 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:34.734290 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:34.734293 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:34.734539 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:34.735017 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.735022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.735697 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.735698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.736494 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.736496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.736806 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.736807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.736900 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.737396 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:34.737404 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:34.738930 (+ 1526us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:34.740648 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:34.740655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:34.740657 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:34.740877 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:34.741091 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:34.741092 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:34.741092 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:34.741202 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:34.741463 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:34.741470 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:34.741603 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 23:03:34.741712 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 23:03:34.741869 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
I20260810 23:03:34.742542 28840 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33931
I20260810 23:03:34.742611 28840 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:34.742775 28840 heartbeater.cc:511] Master 127.23.56.126:33931 requested a full tablet report, sending...
I20260810 23:03:34.743218 28326 ts_manager.cc:194] Registered new tserver with Master: 07b0cccde8ed4698aa2003a58cd534a9 (127.23.56.67:46231)
I20260810 23:03:34.743539 28326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:38483
I20260810 23:03:34.750290 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:03:34.756850 28377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.751538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60500 (local address 127.23.56.126:33931)
0810 23:03:34.751572 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.751574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.751693 (+ 119us) server_negotiation.cc:407] Connection header received
0810 23:03:34.751717 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.751717 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.751726 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.751770 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:34.752249 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.752689 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.753369 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.753559 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.753625 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.754700 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:34.754701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:34.754701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:34.754705 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:34.755911 (+ 1206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.756199 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.756199 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.756199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.756245 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:34.756412 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:34.756413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:34.756413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:34.756540 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:34.756610 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.756698 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.756753 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:03:34.758438 28326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:34.759987 28326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:34.766458 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.762260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40182 (local address 127.23.56.65:41255)
0810 23:03:34.762470 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.762474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.762576 (+ 102us) server_negotiation.cc:407] Connection header received
0810 23:03:34.762630 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.762633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.762650 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.762739 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:34.763198 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.764252 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.765398 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.766103 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.766180 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.766235 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.766332 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.766369 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":88,"threads_started":1}
I20260810 23:03:34.766479 28855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.762121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:41255 (local address 127.0.0.1:40182)
0810 23:03:34.762498 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.762507 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:34.762560 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:34.762761 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:34.762766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:34.762768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:34.763113 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.763115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.764326 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.764327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.765304 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.765306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.766187 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.766188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.766251 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.766261 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:03:34.766337 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:03:34.766388 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":76,"threads_started":1}
I20260810 23:03:34.766598 28852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.762080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:46231 (local address 127.0.0.1:56594)
0810 23:03:34.762266 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.762306 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:34.762372 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:34.762834 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:34.762835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:34.762838 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:34.762922 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.762923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.764247 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.764248 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.765526 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.765529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.766360 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.766361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.766420 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.766423 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:34.766473 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 23:03:34.766524 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"mutex_wait_us":57,"thread_start_us":77,"threads_started":1}
I20260810 23:03:34.766773 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.762149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56594 (local address 127.23.56.67:46231)
0810 23:03:34.762663 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.762666 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.762673 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:03:34.762702 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.762724 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.762740 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.762821 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:34.763001 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.764124 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.765614 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.766283 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.766348 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.766406 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.766693 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.766723 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":76,"threads_started":1}
I20260810 23:03:34.767345 28467 tablet_service.cc:1467] Processing CreateTablet for tablet 795370117d724eb9ab5fa20d14fa780f (DEFAULT_TABLE table=test-table [id=601f5188cdd94bdb8ccc8132a9f9ffa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:34.767443 28767 tablet_service.cc:1467] Processing CreateTablet for tablet 795370117d724eb9ab5fa20d14fa780f (DEFAULT_TABLE table=test-table [id=601f5188cdd94bdb8ccc8132a9f9ffa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:34.767733 28467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795370117d724eb9ab5fa20d14fa780f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:34.767745 28767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795370117d724eb9ab5fa20d14fa780f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:34.768170 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.762295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:34763 (local address 127.0.0.1:50788)
0810 23:03:34.762912 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.762923 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:34.762972 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:34.763284 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:34.763285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:34.763287 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:34.763327 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.763328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.764180 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.764181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.765526 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.765529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.767901 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.767902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.767973 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.767978 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:03:34.768052 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:03:34.768101 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":103,"thread_start_us":67,"threads_started":1}
I20260810 23:03:34.768200 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.762416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.23.56.66:34763)
0810 23:03:34.762633 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.762637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.763043 (+ 406us) server_negotiation.cc:407] Connection header received
0810 23:03:34.763089 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.763091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.763107 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.763196 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:34.763400 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.764077 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.766443 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.767822 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.767901 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.767954 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.768068 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.768116 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":108,"threads_started":1}
I20260810 23:03:34.769042 28615 tablet_service.cc:1467] Processing CreateTablet for tablet 795370117d724eb9ab5fa20d14fa780f (DEFAULT_TABLE table=test-table [id=601f5188cdd94bdb8ccc8132a9f9ffa2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:34.769459 28615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 795370117d724eb9ab5fa20d14fa780f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:34.770721 28857 tablet_bootstrap.cc:492] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f: Bootstrap starting.
I20260810 23:03:34.771106 28858 tablet_bootstrap.cc:492] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: Bootstrap starting.
I20260810 23:03:34.771438 28857 tablet_bootstrap.cc:654] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:34.771778 28858 tablet_bootstrap.cc:654] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:34.772069 28857 log.cc:824] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:34.772413 28858 log.cc:824] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:34.773118 28858 tablet_bootstrap.cc:492] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: No bootstrap required, opened a new log
I20260810 23:03:34.773182 28858 ts_tablet_manager.cc:1397] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:03:34.773553 28857 tablet_bootstrap.cc:492] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f: No bootstrap required, opened a new log
I20260810 23:03:34.773628 28857 ts_tablet_manager.cc:1397] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:03:34.774291 28861 tablet_bootstrap.cc:492] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9: Bootstrap starting.
I20260810 23:03:34.775247 28858 raft_consensus.cc:348] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } }
I20260810 23:03:34.775368 28861 tablet_bootstrap.cc:654] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:34.775408 28858 raft_consensus.cc:374] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:34.775445 28858 raft_consensus.cc:729] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07b0cccde8ed4698aa2003a58cd534a9, State: Initialized, Role: FOLLOWER
I20260810 23:03:34.775563 28858 consensus_queue.cc:260] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } }
I20260810 23:03:34.775674 28857 raft_consensus.cc:348] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } }
I20260810 23:03:34.775856 28858 ts_tablet_manager.cc:1428] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 23:03:34.775846 28857 raft_consensus.cc:374] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:34.775910 28857 raft_consensus.cc:729] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccadff7636fc4731ad02e65f9a0f000f, State: Initialized, Role: FOLLOWER
I20260810 23:03:34.776026 28857 consensus_queue.cc:260] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } }
I20260810 23:03:34.776311 28857 ts_tablet_manager.cc:1428] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:34.776350 28542 heartbeater.cc:503] Master 127.23.56.126:33931 was elected leader, sending a full tablet report...
I20260810 23:03:34.776423 28840 heartbeater.cc:503] Master 127.23.56.126:33931 was elected leader, sending a full tablet report...
I20260810 23:03:34.776538 28861 log.cc:824] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:34.777514 28861 tablet_bootstrap.cc:492] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9: No bootstrap required, opened a new log
I20260810 23:03:34.777582 28861 ts_tablet_manager.cc:1397] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:03:34.778867 28861 raft_consensus.cc:348] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } }
I20260810 23:03:34.778983 28861 raft_consensus.cc:374] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:34.779007 28861 raft_consensus.cc:729] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 416506d026894fbfaa7f6062d01b56e9, State: Initialized, Role: FOLLOWER
I20260810 23:03:34.779112 28861 consensus_queue.cc:260] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } }
I20260810 23:03:34.779371 28861 ts_tablet_manager.cc:1428] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:34.779485 28693 heartbeater.cc:503] Master 127.23.56.126:33931 was elected leader, sending a full tablet report...
W20260810 23:03:34.822052 28694 tablet.cc:2369] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:34.901889 28543 tablet.cc:2369] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:03:34.910830 28865 raft_consensus.cc:482] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:34.910945 28865 raft_consensus.cc:504] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } }
I20260810 23:03:34.911366 28865 leader_election.cc:288] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 07b0cccde8ed4698aa2003a58cd534a9 (127.23.56.67:46231), ccadff7636fc4731ad02e65f9a0f000f (127.23.56.65:41255)
I20260810 23:03:34.914340 28854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.911638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:51469 (local address 127.23.56.65:41255)
0810 23:03:34.911709 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.911712 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.911962 (+ 250us) server_negotiation.cc:407] Connection header received
0810 23:03:34.911999 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.912000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.912012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.912064 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:34.912216 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.912678 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.913641 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.913986 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.914082 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.914090 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.914225 (+ 135us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.914270 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:03:34.914352 28866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.911541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:41255 (local address 127.23.56.66:51469)
0810 23:03:34.911872 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.911881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:34.911930 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:34.912079 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:34.912080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:34.912082 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:34.912148 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.912149 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.912756 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.912757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.913520 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.913522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.914083 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.914084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.914156 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.914159 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:34.914201 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 23:03:34.914231 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":125,"thread_start_us":145,"threads_started":1}
I20260810 23:03:34.914398 28853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.911637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:56759 (local address 127.23.56.67:46231)
0810 23:03:34.911703 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.911705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.911717 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:03:34.911747 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.911748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.911765 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.911852 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:34.912071 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.912610 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.913607 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.913999 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.914082 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.914090 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.914281 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.914322 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:03:34.914357 28695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.911541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:46231 (local address 127.23.56.66:56759)
0810 23:03:34.911603 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.911616 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:34.911658 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:34.911860 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:34.911862 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:34.911864 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:34.911961 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.911965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.912696 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.912697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.913509 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:34.913511 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.914084 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.914085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:34.914182 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.914202 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 23:03:34.914259 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 23:03:34.914290 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":29}
I20260810 23:03:34.914767 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795370117d724eb9ab5fa20d14fa780f" candidate_uuid: "416506d026894fbfaa7f6062d01b56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccadff7636fc4731ad02e65f9a0f000f" is_pre_election: true
I20260810 23:03:34.914896 28788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795370117d724eb9ab5fa20d14fa780f" candidate_uuid: "416506d026894fbfaa7f6062d01b56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b0cccde8ed4698aa2003a58cd534a9" is_pre_election: true
I20260810 23:03:34.914938 28489 raft_consensus.cc:2454] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416506d026894fbfaa7f6062d01b56e9 in term 0.
I20260810 23:03:34.915032 28788 raft_consensus.cc:2454] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 416506d026894fbfaa7f6062d01b56e9 in term 0.
I20260810 23:03:34.915215 28567 leader_election.cc:302] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 416506d026894fbfaa7f6062d01b56e9, ccadff7636fc4731ad02e65f9a0f000f; no voters:
I20260810 23:03:34.915330 28865 raft_consensus.cc:2781] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:34.915385 28865 raft_consensus.cc:482] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:34.915407 28865 raft_consensus.cc:3037] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:34.916046 28865 raft_consensus.cc:504] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } }
I20260810 23:03:34.916190 28865 leader_election.cc:288] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [CANDIDATE]: Term 1 election: Requested vote from peers 07b0cccde8ed4698aa2003a58cd534a9 (127.23.56.67:46231), ccadff7636fc4731ad02e65f9a0f000f (127.23.56.65:41255)
I20260810 23:03:34.916373 28788 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795370117d724eb9ab5fa20d14fa780f" candidate_uuid: "416506d026894fbfaa7f6062d01b56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "07b0cccde8ed4698aa2003a58cd534a9"
I20260810 23:03:34.916460 28788 raft_consensus.cc:3037] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:34.917223 28788 raft_consensus.cc:2454] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416506d026894fbfaa7f6062d01b56e9 in term 1.
I20260810 23:03:34.917366 28489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "795370117d724eb9ab5fa20d14fa780f" candidate_uuid: "416506d026894fbfaa7f6062d01b56e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccadff7636fc4731ad02e65f9a0f000f"
I20260810 23:03:34.917418 28568 leader_election.cc:302] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 07b0cccde8ed4698aa2003a58cd534a9, 416506d026894fbfaa7f6062d01b56e9; no voters:
I20260810 23:03:34.917454 28489 raft_consensus.cc:3037] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:34.917548 28865 raft_consensus.cc:2781] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:34.917717 28865 raft_consensus.cc:686] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 1 LEADER]: Becoming Leader. State: Replica: 416506d026894fbfaa7f6062d01b56e9, State: Running, Role: LEADER
I20260810 23:03:34.917821 28865 consensus_queue.cc:237] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } }
I20260810 23:03:34.918298 28489 raft_consensus.cc:2454] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 416506d026894fbfaa7f6062d01b56e9 in term 1.
I20260810 23:03:34.918669 28326 catalog_manager.cc:5495] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 416506d026894fbfaa7f6062d01b56e9 (127.23.56.66). New cstate: current_term: 1 leader_uuid: "416506d026894fbfaa7f6062d01b56e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "416506d026894fbfaa7f6062d01b56e9" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34763 } health_report { overall_health: HEALTHY } } }
I20260810 23:03:34.946902 28856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.944456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.23.56.66:34763)
0810 23:03:34.944509 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:34.944511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:34.944553 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:03:34.944590 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:34.944591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:34.944615 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:34.944711 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:34.944875 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.945344 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.946095 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:34.946294 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:34.946371 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:34.946555 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:34.946657 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:34.946753 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:34.946771 (+ 18us) server_negotiation.cc:299] Negotiation successful
0810 23:03:34.946791 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 23:03:34.948674 28788 raft_consensus.cc:1261] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [term 1 FOLLOWER]: Refusing update from remote peer 416506d026894fbfaa7f6062d01b56e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:34.948716 28489 raft_consensus.cc:1261] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [term 1 FOLLOWER]: Refusing update from remote peer 416506d026894fbfaa7f6062d01b56e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:34.949075 28865 consensus_queue.cc:1035] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccadff7636fc4731ad02e65f9a0f000f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:34.949215 28867 consensus_queue.cc:1035] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "07b0cccde8ed4698aa2003a58cd534a9" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:34.951930 28870 mvcc.cc:205] Tried to move back new op lower bound from 7317106749227282432 to 7317106749104025600. Current Snapshot: MvccSnapshot[applied={T|T < 7317106749227282432}]
I20260810 23:03:34.954016 28326 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:34.954113 28326 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:34.954854 28326 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 74d8d5f097c14241b02c8846fe96463f: Sending DeleteTablet for 3 replicas of tablet 795370117d724eb9ab5fa20d14fa780f
I20260810 23:03:34.955238 28467 tablet_service.cc:1514] Processing DeleteTablet for tablet 795370117d724eb9ab5fa20d14fa780f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:34 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:40182
I20260810 23:03:34.955259 28615 tablet_service.cc:1514] Processing DeleteTablet for tablet 795370117d724eb9ab5fa20d14fa780f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:34 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:50788
I20260810 23:03:34.955282 28767 tablet_service.cc:1514] Processing DeleteTablet for tablet 795370117d724eb9ab5fa20d14fa780f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:34 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:56594
I20260810 23:03:34.955509 28881 tablet_replica.cc:330] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f: stopping tablet replica
I20260810 23:03:34.955611 28881 raft_consensus.cc:2229] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:34.955659 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 28401
I20260810 23:03:34.955725 28881 raft_consensus.cc:2258] T 795370117d724eb9ab5fa20d14fa780f P ccadff7636fc4731ad02e65f9a0f000f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:34.963245 28883 tablet_replica.cc:330] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: stopping tablet replica
I20260810 23:03:34.963375 28883 raft_consensus.cc:2229] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:34.963486 28883 raft_consensus.cc:2258] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:34.963740 28882 tablet_replica.cc:330] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9: stopping tablet replica
I20260810 23:03:34.963861 28882 raft_consensus.cc:2229] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:34.963987 28882 raft_consensus.cc:2258] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 23:03:34.964206 28312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:41255 (error 108)
I20260810 23:03:34.964447 28882 ts_tablet_manager.cc:1905] T 795370117d724eb9ab5fa20d14fa780f P 416506d026894fbfaa7f6062d01b56e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:34.964836 28883 ts_tablet_manager.cc:1905] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:03:34.964859 28851 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:34.964606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:41255 (local address 127.0.0.1:40198)
0810 23:03:34.964654 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.964664 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:34.964702 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:34.964813 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:41255: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 23:03:34.964968 28312 catalog_manager.cc:4584] TS ccadff7636fc4731ad02e65f9a0f000f (127.23.56.65:41255): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 795370117d724eb9ab5fa20d14fa780f: Network error: Client connection negotiation failed: client connection to 127.23.56.65:41255: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:03:34.965397 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 28547
I20260810 23:03:34.966465 28883 ts_tablet_manager.cc:1918] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:34.966542 28883 log.cc:1196] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/795370117d724eb9ab5fa20d14fa780f
I20260810 23:03:34.966810 28883 ts_tablet_manager.cc:1939] T 795370117d724eb9ab5fa20d14fa780f P 07b0cccde8ed4698aa2003a58cd534a9: Deleting consensus metadata
I20260810 23:03:34.967249 28313 catalog_manager.cc:4841] TS 07b0cccde8ed4698aa2003a58cd534a9 (127.23.56.67:46231): tablet 795370117d724eb9ab5fa20d14fa780f (table test-table [id=601f5188cdd94bdb8ccc8132a9f9ffa2]) successfully deleted
I20260810 23:03:34.971400 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 28698
I20260810 23:03:34.971477 28851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:34.971315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:34763 (local address 127.0.0.1:50802)
0810 23:03:34.971375 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:34.971424 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.66:34763: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 23:03:34.983103 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 28295
2026-08-10T23:03:34Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2119 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T23:03:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:35Z Disabled control of system clock
I20260810 23:03:35.013350 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45967
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:45967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:03:35.095587 28888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:35.095781 28888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:35.095801 28888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:35.096920 28888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:35.096956 28888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:35.096969 28888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:35.096982 28888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:35.096994 28888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:35.098351 28888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:45967
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:35.098541 28888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:35.098780 28888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:35.103753 28895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:35.103937 28888 server_base.cc:1034] running on GCE node
W20260810 23:03:35.103804 28894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:35.103753 28897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:35.104250 28888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:35.104569 28888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:35.105728 28888 hybrid_clock.cc:648] HybridClock initialized: now 1786403015105709 us; error 43 us; skew 500 ppm
I20260810 23:03:35.107012 28888 webserver.cc:460] Webserver started at http://127.23.56.126:45419/ using document root <none> and password file <none>
I20260810 23:03:35.107337 28888 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:35.107409 28888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:35.107587 28888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:35.108731 28888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "9e3dd344741449eba8679cfefdc0214b"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "420cb13f23ba140bad798210ecb24316"
server_key_iv: "48f778aef5caea695ff865bc719144dd"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.109122 28888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "9e3dd344741449eba8679cfefdc0214b"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "420cb13f23ba140bad798210ecb24316"
server_key_iv: "48f778aef5caea695ff865bc719144dd"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.110621 28888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:35.111438 28904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.111619 28888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:35.111728 28888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "9e3dd344741449eba8679cfefdc0214b"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "420cb13f23ba140bad798210ecb24316"
server_key_iv: "48f778aef5caea695ff865bc719144dd"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.111804 28888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:35.115226 28888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:35.115530 28888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:35.115679 28888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:35.119699 28888 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:45967
I20260810 23:03:35.119735 28959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:45967 every 8 connection(s)
I20260810 23:03:35.120077 28888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:35.120723 28960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:35.123453 28960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b: Bootstrap starting.
I20260810 23:03:35.124043 28960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:35.124336 28960 log.cc:824] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:35.124989 28960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b: No bootstrap required, opened a new log
I20260810 23:03:35.126525 28960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd344741449eba8679cfefdc0214b" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45967 } }
I20260810 23:03:35.126653 28960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:35.126675 28960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3dd344741449eba8679cfefdc0214b, State: Initialized, Role: FOLLOWER
I20260810 23:03:35.126835 28960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd344741449eba8679cfefdc0214b" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45967 } }
I20260810 23:03:35.126899 28960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:35.126937 28960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:35.126988 28960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:35.127573 28960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd344741449eba8679cfefdc0214b" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45967 } }
I20260810 23:03:35.127692 28960 leader_election.cc:302] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e3dd344741449eba8679cfefdc0214b; no voters:
I20260810 23:03:35.127749 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 28888
I20260810 23:03:35.127864 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:35.127866 28960 leader_election.cc:288] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:35.127897 28964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:35.128026 28964 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [term 1 LEADER]: Becoming Leader. State: Replica: 9e3dd344741449eba8679cfefdc0214b, State: Running, Role: LEADER
I20260810 23:03:35.128077 23777 external_mini_cluster.cc:1428] Setting key 68269b1509903e218753a83ac698693c
I20260810 23:03:35.128124 28960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:35.128104 28964 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd344741449eba8679cfefdc0214b" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45967 } }
I20260810 23:03:35.128468 28964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e3dd344741449eba8679cfefdc0214b. Latest consensus state: current_term: 1 leader_uuid: "9e3dd344741449eba8679cfefdc0214b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd344741449eba8679cfefdc0214b" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45967 } } }
I20260810 23:03:35.128538 28964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:35.128705 28965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e3dd344741449eba8679cfefdc0214b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e3dd344741449eba8679cfefdc0214b" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45967 } } }
I20260810 23:03:35.128796 28965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:35.129050 28970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:35.129827 28970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:35.130812 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.128962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.23.56.126:45967)
0810 23:03:35.129178 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.129187 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.129220 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:03:35.129341 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.129349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.129391 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.129549 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.130301 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.130305 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.130305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.130311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.130355 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.130369 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.130440 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.130547 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.130615 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":86,"threads_started":1}
I20260810 23:03:35.131754 28970 catalog_manager.cc:1347] Generated new cluster ID: 7d556cf374d247d09e780823d5c1b145
I20260810 23:03:35.131806 28970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:35.143915 28970 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:35.144097 28970 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:35.153323 28970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b: Generated new TSK 0
I20260810 23:03:35.153522 28970 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:03:35.155774 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45967
--builtin_ntp_servers=127.23.56.84:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:03:35.240666 28987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:35.240870 28987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:35.240892 28987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:35.242044 28987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:35.242081 28987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:35.242105 28987 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:35.243471 28987 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28987
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:35.243682 28987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:35.244014 28987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:35.249166 28996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:35.249191 28992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:35.249177 28993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:35.249716 28987 server_base.cc:1034] running on GCE node
I20260810 23:03:35.249963 28987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:35.250242 28987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:35.251458 28987 hybrid_clock.cc:648] HybridClock initialized: now 1786403015251436 us; error 38 us; skew 500 ppm
I20260810 23:03:35.252972 28987 webserver.cc:460] Webserver started at http://127.23.56.65:36445/ using document root <none> and password file <none>
I20260810 23:03:35.253227 28987 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:35.253293 28987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:35.253429 28987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:35.254834 28987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "70f1ee41140243d299a682573343f101"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "5a0a689a9d22205d9bfb523e1697e5af"
server_key_iv: "0b2862325d0dea57c26d12bfbfe90427"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.255271 28987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "70f1ee41140243d299a682573343f101"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "5a0a689a9d22205d9bfb523e1697e5af"
server_key_iv: "0b2862325d0dea57c26d12bfbfe90427"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.256968 28987 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 23:03:35.257858 29002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.258044 28987 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:35.258134 28987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "70f1ee41140243d299a682573343f101"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "5a0a689a9d22205d9bfb523e1697e5af"
server_key_iv: "0b2862325d0dea57c26d12bfbfe90427"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.258206 28987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:35.261395 28987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:35.261650 28987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:35.261790 28987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:35.262039 28987 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:35.262342 28987 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:35.262375 28987 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.262414 28987 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:35.262439 28987 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.267961 28987 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:34335
I20260810 23:03:35.268116 29127 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:34335 every 8 connection(s)
I20260810 23:03:35.268347 28987 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:35.271454 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.269094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:33587 (local address 127.23.56.126:45967)
0810 23:03:35.269134 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.269136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.269352 (+ 216us) server_negotiation.cc:407] Connection header received
0810 23:03:35.269807 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.269808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.269816 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.269891 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.271173 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.271174 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.271175 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.271180 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.271202 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.271208 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.271306 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.271355 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.271370 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:03:35.271610 29130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.269083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45967 (local address 127.23.56.65:33587)
0810 23:03:35.269247 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:35.269277 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:35.269711 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:35.269938 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:35.269941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:35.269948 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:35.270984 (+ 1036us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:03:35.270985 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:35.271014 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:03:35.271042 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:35.271300 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:35.271304 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:35.271306 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:03:35.271360 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 23:03:35.271407 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":83,"threads_started":1}
I20260810 23:03:35.272037 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 28987
I20260810 23:03:35.272128 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:35.272215 29128 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45967
I20260810 23:03:35.272287 23777 external_mini_cluster.cc:1428] Setting key 702042b0b7080a77b1d178143cbdcf85
I20260810 23:03:35.272380 29128 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:35.272825 29128 heartbeater.cc:511] Master 127.23.56.126:45967 requested a full tablet report, sending...
I20260810 23:03:35.273274 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45967
--builtin_ntp_servers=127.23.56.84:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 23:03:35.273458 28921 ts_manager.cc:194] Registered new tserver with Master: 70f1ee41140243d299a682573343f101 (127.23.56.65:34335)
W20260810 23:03:35.358187 29131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:35.358451 29131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:35.358479 29131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:35.359634 29131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:35.359669 29131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:35.359695 29131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:35.361039 29131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:35.361230 29131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:35.361522 29131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:35.366684 29136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:35.366727 29139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:35.366788 29131 server_base.cc:1034] running on GCE node
W20260810 23:03:35.366667 29137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:35.367084 29131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:35.367373 29131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:35.368534 29131 hybrid_clock.cc:648] HybridClock initialized: now 1786403015368510 us; error 32 us; skew 500 ppm
I20260810 23:03:35.369853 29131 webserver.cc:460] Webserver started at http://127.23.56.66:43539/ using document root <none> and password file <none>
I20260810 23:03:35.370095 29131 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:35.370141 29131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:35.370309 29131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:35.371510 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "a6cdd76859604e969f2372733352e8ec"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "0ed53d56ac9fb283b826eb55992e960c"
server_key_iv: "41333c077bdd64bfb9d89bd56dad60a9"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.371857 29131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "a6cdd76859604e969f2372733352e8ec"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "0ed53d56ac9fb283b826eb55992e960c"
server_key_iv: "41333c077bdd64bfb9d89bd56dad60a9"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.373132 29131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:35.373901 29146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.374081 29131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:35.374159 29131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "a6cdd76859604e969f2372733352e8ec"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "0ed53d56ac9fb283b826eb55992e960c"
server_key_iv: "41333c077bdd64bfb9d89bd56dad60a9"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.374210 29131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:35.377389 29131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:35.377617 29131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:35.377712 29131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:35.377856 29131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:35.378091 29131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:35.378130 29131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.378149 29131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:35.378162 29131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.383720 29131 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:41219
I20260810 23:03:35.383827 29266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:41219 every 8 connection(s)
I20260810 23:03:35.384087 29131 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:35.386987 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.384911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:59299 (local address 127.23.56.126:45967)
0810 23:03:35.384994 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.384997 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.385174 (+ 177us) server_negotiation.cc:407] Connection header received
0810 23:03:35.385592 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.385594 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.385600 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.385662 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.386744 (+ 1082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.386746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.386746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.386750 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.386766 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.386771 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.386834 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.386919 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.386933 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 23:03:35.387102 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.384872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45967 (local address 127.23.56.66:59299)
0810 23:03:35.385048 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:35.385085 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:35.385504 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:35.385735 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:35.385738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:35.385746 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:35.386607 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:03:35.386608 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:35.386634 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:03:35.386659 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:35.386832 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:35.386835 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:35.386837 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:03:35.386886 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 23:03:35.386930 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
I20260810 23:03:35.387631 29267 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45967
I20260810 23:03:35.387830 29267 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:35.387996 29267 heartbeater.cc:511] Master 127.23.56.126:45967 requested a full tablet report, sending...
I20260810 23:03:35.388370 28921 ts_manager.cc:194] Registered new tserver with Master: a6cdd76859604e969f2372733352e8ec (127.23.56.66:41219)
I20260810 23:03:35.388906 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 29131
I20260810 23:03:35.389014 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:35.389195 23777 external_mini_cluster.cc:1428] Setting key 24ff177c86b598a9920cc17fb304bc26
I20260810 23:03:35.390007 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45967
--builtin_ntp_servers=127.23.56.84:42197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 23:03:35.471403 29271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:35.471616 29271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:35.471652 29271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:35.472780 29271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:35.472851 29271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:35.472894 29271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:35.474285 29271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45967
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:35.474548 29271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:35.474848 29271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:35.480023 29279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:35.480095 29276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:35.480095 29277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:35.480218 29271 server_base.cc:1034] running on GCE node
I20260810 23:03:35.480535 29271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:35.480736 29271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:35.481868 29271 hybrid_clock.cc:648] HybridClock initialized: now 1786403015481861 us; error 31 us; skew 500 ppm
I20260810 23:03:35.483094 29271 webserver.cc:460] Webserver started at http://127.23.56.67:44055/ using document root <none> and password file <none>
I20260810 23:03:35.483386 29271 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:35.483433 29271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:35.483595 29271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:35.484645 29271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "fb391ba4413d4cb89b6ab44f2a506d2b"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "41fd0a6e8730d1927fd5120ede7d44da"
server_key_iv: "e7d2e5aefb922019d12863d65cdac135"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.484983 29271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "fb391ba4413d4cb89b6ab44f2a506d2b"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "41fd0a6e8730d1927fd5120ede7d44da"
server_key_iv: "e7d2e5aefb922019d12863d65cdac135"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.486227 29271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:35.486948 29288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.487174 29271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:35.487255 29271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "fb391ba4413d4cb89b6ab44f2a506d2b"
format_stamp: "Formatted at 2026-08-10 23:03:35 on dist-test-slave-1xrh"
server_key: "41fd0a6e8730d1927fd5120ede7d44da"
server_key_iv: "e7d2e5aefb922019d12863d65cdac135"
server_key_version: "encryptionkey@0"
I20260810 23:03:35.487308 29271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:35.490409 29271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:35.490625 29271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:35.490716 29271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:35.490867 29271 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:35.491099 29271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:35.491195 29271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.491227 29271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:35.491245 29271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:35.496575 29271 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:40533
I20260810 23:03:35.496650 29408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:40533 every 8 connection(s)
I20260810 23:03:35.496954 29271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:35.499847 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.497655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:34195 (local address 127.23.56.126:45967)
0810 23:03:35.497737 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.497739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.497919 (+ 180us) server_negotiation.cc:407] Connection header received
0810 23:03:35.498326 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.498326 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.498332 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.498379 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.499578 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.499580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.499580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.499584 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.499598 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.499602 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.499650 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.499725 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.499743 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:03:35.499924 29411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.497608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45967 (local address 127.23.56.67:34195)
0810 23:03:35.497799 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:35.497838 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:35.498242 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:35.498436 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:35.498439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:35.498446 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:35.499440 (+ 994us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:03:35.499441 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:35.499466 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:03:35.499492 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:35.499648 (+ 156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:35.499650 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:35.499652 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:03:35.499700 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 23:03:35.499742 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260810 23:03:35.500392 29409 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45967
I20260810 23:03:35.500474 29409 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:35.500626 29409 heartbeater.cc:511] Master 127.23.56.126:45967 requested a full tablet report, sending...
I20260810 23:03:35.500988 28921 ts_manager.cc:194] Registered new tserver with Master: fb391ba4413d4cb89b6ab44f2a506d2b (127.23.56.67:40533)
I20260810 23:03:35.505041 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 29271
I20260810 23:03:35.505136 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:35.505308 23777 external_mini_cluster.cc:1428] Setting key 6bd72044ad1afbb855ff3824f4576ef0
I20260810 23:03:35.506129 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:03:35.508476 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.507445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53868 (local address 127.23.56.126:45967)
0810 23:03:35.507482 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.507484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.507613 (+ 129us) server_negotiation.cc:407] Connection header received
0810 23:03:35.507693 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.507694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.507699 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.507754 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.508258 (+ 504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.508259 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.508260 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.508264 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.508278 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.508282 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.508333 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.508391 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.508407 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:03:35.509969 28921 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:35.511865 28921 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:35.515722 29417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.513911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:40533 (local address 127.0.0.1:53228)
0810 23:03:35.514045 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:35.514085 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:35.514144 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:35.514812 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:35.514813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:35.514815 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:35.515258 (+ 443us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:03:35.515258 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:35.515273 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:03:35.515298 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:35.515528 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:35.515534 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:35.515537 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:35.515620 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:03:35.515634 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":84,"threads_started":1}
I20260810 23:03:35.515748 29418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.513992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53228 (local address 127.23.56.67:40533)
0810 23:03:35.514638 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.514665 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.514675 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:35.514696 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.514699 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.514706 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.514953 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.515365 (+ 412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.515371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.515371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.515375 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.515427 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.515445 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.515525 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.515626 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.515642 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":80,"threads_started":1}
I20260810 23:03:35.515952 29419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.514152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:41219 (local address 127.0.0.1:43368)
0810 23:03:35.514461 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:35.514471 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:35.514528 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:35.514692 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:35.514698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:35.514704 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:35.515455 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:03:35.515455 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:35.515462 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:03:35.515474 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:35.515725 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:35.515726 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:35.515726 (+ 0us) client_negotiation.cc:769] Sending connection context
0810 23:03:35.515768 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 23:03:35.515781 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":77,"threads_started":1}
I20260810 23:03:35.516242 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.514261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43368 (local address 127.23.56.66:41219)
0810 23:03:35.514447 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.514452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.514545 (+ 93us) server_negotiation.cc:407] Connection header received
0810 23:03:35.514578 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.514581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.514591 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.514681 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.515562 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.515569 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.515569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.515573 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.515609 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.515635 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.515688 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.516142 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.516160 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":83,"threads_started":1}
I20260810 23:03:35.516268 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.513976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:34335 (local address 127.0.0.1:39520)
0810 23:03:35.514460 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:35.514471 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:35.514535 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:35.515102 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:35.515103 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:35.515106 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:35.515851 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:03:35.515851 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:35.515857 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:03:35.515869 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:35.516149 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:35.516150 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:35.516151 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 23:03:35.516194 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 23:03:35.516206 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":109,"threads_started":1}
I20260810 23:03:35.516537 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.514229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39520 (local address 127.23.56.65:34335)
0810 23:03:35.514447 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.514452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.514552 (+ 100us) server_negotiation.cc:407] Connection header received
0810 23:03:35.514583 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.514585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.514592 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.514680 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.515952 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.515959 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.515959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.515963 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.516010 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.516061 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.516123 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.516439 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.516456 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":110,"threads_started":1}
I20260810 23:03:35.516517 29338 tablet_service.cc:1467] Processing CreateTablet for tablet bc84442a1ec343c68f477c5824436bb4 (DEFAULT_TABLE table=test-table [id=dafa6324dbf0480e97a226a495d2d0a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:35.516914 29338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc84442a1ec343c68f477c5824436bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:35.516959 29198 tablet_service.cc:1467] Processing CreateTablet for tablet bc84442a1ec343c68f477c5824436bb4 (DEFAULT_TABLE table=test-table [id=dafa6324dbf0480e97a226a495d2d0a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:35.517164 29056 tablet_service.cc:1467] Processing CreateTablet for tablet bc84442a1ec343c68f477c5824436bb4 (DEFAULT_TABLE table=test-table [id=dafa6324dbf0480e97a226a495d2d0a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:35.517333 29198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc84442a1ec343c68f477c5824436bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:35.517478 29056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc84442a1ec343c68f477c5824436bb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:35.519675 29424 tablet_bootstrap.cc:492] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: Bootstrap starting.
I20260810 23:03:35.520282 29424 tablet_bootstrap.cc:654] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:35.520646 29426 tablet_bootstrap.cc:492] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101: Bootstrap starting.
I20260810 23:03:35.520821 29424 log.cc:824] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:35.520864 29425 tablet_bootstrap.cc:492] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: Bootstrap starting.
I20260810 23:03:35.521461 29425 tablet_bootstrap.cc:654] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:35.521756 29424 tablet_bootstrap.cc:492] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: No bootstrap required, opened a new log
I20260810 23:03:35.521783 29425 log.cc:824] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:35.521822 29424 ts_tablet_manager.cc:1397] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:03:35.522588 29426 tablet_bootstrap.cc:654] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:35.522895 29426 log.cc:824] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:35.523443 29425 tablet_bootstrap.cc:492] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: No bootstrap required, opened a new log
I20260810 23:03:35.523521 29425 ts_tablet_manager.cc:1397] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:03:35.523708 29426 tablet_bootstrap.cc:492] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101: No bootstrap required, opened a new log
I20260810 23:03:35.523805 29426 ts_tablet_manager.cc:1397] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:03:35.524539 29424 raft_consensus.cc:348] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } }
I20260810 23:03:35.524700 29424 raft_consensus.cc:374] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:35.524744 29424 raft_consensus.cc:729] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb391ba4413d4cb89b6ab44f2a506d2b, State: Initialized, Role: FOLLOWER
I20260810 23:03:35.524873 29424 consensus_queue.cc:260] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } }
I20260810 23:03:35.525132 29424 ts_tablet_manager.cc:1428] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:35.525250 29409 heartbeater.cc:503] Master 127.23.56.126:45967 was elected leader, sending a full tablet report...
I20260810 23:03:35.526106 29425 raft_consensus.cc:348] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } }
I20260810 23:03:35.526263 29425 raft_consensus.cc:374] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:35.526289 29425 raft_consensus.cc:729] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6cdd76859604e969f2372733352e8ec, State: Initialized, Role: FOLLOWER
I20260810 23:03:35.526409 29425 consensus_queue.cc:260] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } }
I20260810 23:03:35.526544 29426 raft_consensus.cc:348] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } }
I20260810 23:03:35.526692 29267 heartbeater.cc:503] Master 127.23.56.126:45967 was elected leader, sending a full tablet report...
I20260810 23:03:35.526702 29426 raft_consensus.cc:374] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:35.526747 29426 raft_consensus.cc:729] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70f1ee41140243d299a682573343f101, State: Initialized, Role: FOLLOWER
I20260810 23:03:35.526680 29425 ts_tablet_manager.cc:1428] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:35.526866 29426 consensus_queue.cc:260] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } }
I20260810 23:03:35.527122 29426 ts_tablet_manager.cc:1428] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 23:03:35.527292 29128 heartbeater.cc:503] Master 127.23.56.126:45967 was elected leader, sending a full tablet report...
W20260810 23:03:35.634635 29269 tablet.cc:2369] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:35.747499 29410 tablet.cc:2369] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:35.769119 29129 tablet.cc:2369] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:03:35.821391 29431 raft_consensus.cc:482] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:35.821542 29431 raft_consensus.cc:504] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } }
I20260810 23:03:35.821902 29431 leader_election.cc:288] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb391ba4413d4cb89b6ab44f2a506d2b (127.23.56.67:40533), 70f1ee41140243d299a682573343f101 (127.23.56.65:34335)
I20260810 23:03:35.823359 29270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.822119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:34335 (local address 127.23.56.66:39167)
0810 23:03:35.822185 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:35.822197 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:35.822244 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:35.822412 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:35.822414 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:35.822416 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:35.822987 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:03:35.822988 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:35.822997 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:03:35.823009 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:35.823220 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:35.823220 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:35.823221 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 23:03:35.823272 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 23:03:35.823288 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:03:35.823359 29421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.822165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:39167 (local address 127.23.56.65:34335)
0810 23:03:35.822240 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.822242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.822251 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:35.822317 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.822317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.822323 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.822374 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.823099 (+ 725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.823101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.823101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.823106 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.823163 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.823169 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.823222 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.823273 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.823288 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:03:35.823570 29418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.822174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:38303 (local address 127.23.56.67:40533)
0810 23:03:35.822222 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.822224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.822577 (+ 353us) server_negotiation.cc:407] Connection header received
0810 23:03:35.822611 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.822613 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.822619 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.822673 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.823297 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.823298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.823299 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.823304 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.823320 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.823325 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.823403 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.823485 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.823500 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:03:35.823757 29433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.822125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:40533 (local address 127.23.56.66:38303)
0810 23:03:35.822499 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:35.822508 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:35.822553 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:35.822703 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:35.822704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:35.822707 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:35.823178 (+ 471us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:03:35.823179 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:35.823186 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:03:35.823199 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:35.823399 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:35.823399 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:35.823400 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 23:03:35.823667 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 23:03:35.823683 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":184,"thread_start_us":114,"threads_started":1}
I20260810 23:03:35.823721 29077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc84442a1ec343c68f477c5824436bb4" candidate_uuid: "a6cdd76859604e969f2372733352e8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f1ee41140243d299a682573343f101" is_pre_election: true
I20260810 23:03:35.823889 29077 raft_consensus.cc:2454] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6cdd76859604e969f2372733352e8ec in term 0.
I20260810 23:03:35.824156 29148 leader_election.cc:302] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70f1ee41140243d299a682573343f101, a6cdd76859604e969f2372733352e8ec; no voters:
I20260810 23:03:35.824198 29361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc84442a1ec343c68f477c5824436bb4" candidate_uuid: "a6cdd76859604e969f2372733352e8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" is_pre_election: true
I20260810 23:03:35.824285 29431 raft_consensus.cc:2781] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:35.824338 29431 raft_consensus.cc:482] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:35.824370 29431 raft_consensus.cc:3037] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:35.824368 29361 raft_consensus.cc:2454] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a6cdd76859604e969f2372733352e8ec in term 0.
I20260810 23:03:35.825176 29431 raft_consensus.cc:504] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } }
I20260810 23:03:35.825327 29431 leader_election.cc:288] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [CANDIDATE]: Term 1 election: Requested vote from peers fb391ba4413d4cb89b6ab44f2a506d2b (127.23.56.67:40533), 70f1ee41140243d299a682573343f101 (127.23.56.65:34335)
I20260810 23:03:35.825475 29361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc84442a1ec343c68f477c5824436bb4" candidate_uuid: "a6cdd76859604e969f2372733352e8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b"
I20260810 23:03:35.825528 29077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc84442a1ec343c68f477c5824436bb4" candidate_uuid: "a6cdd76859604e969f2372733352e8ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70f1ee41140243d299a682573343f101"
I20260810 23:03:35.825562 29361 raft_consensus.cc:3037] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:35.825639 29077 raft_consensus.cc:3037] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:35.826315 29361 raft_consensus.cc:2454] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6cdd76859604e969f2372733352e8ec in term 1.
I20260810 23:03:35.826502 29077 raft_consensus.cc:2454] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a6cdd76859604e969f2372733352e8ec in term 1.
I20260810 23:03:35.826689 29147 leader_election.cc:302] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a6cdd76859604e969f2372733352e8ec, fb391ba4413d4cb89b6ab44f2a506d2b; no voters:
I20260810 23:03:35.826808 29431 raft_consensus.cc:2781] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:35.826977 29431 raft_consensus.cc:686] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 1 LEADER]: Becoming Leader. State: Replica: a6cdd76859604e969f2372733352e8ec, State: Running, Role: LEADER
I20260810 23:03:35.827072 29431 consensus_queue.cc:237] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } }
I20260810 23:03:35.827975 28920 catalog_manager.cc:5495] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec reported cstate change: term changed from 0 to 1, leader changed from <none> to a6cdd76859604e969f2372733352e8ec (127.23.56.66). New cstate: current_term: 1 leader_uuid: "a6cdd76859604e969f2372733352e8ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6cdd76859604e969f2372733352e8ec" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 41219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 } health_report { overall_health: UNKNOWN } } }
I20260810 23:03:35.870918 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.869806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.23.56.66:41219)
0810 23:03:35.869858 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:35.869860 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:35.869869 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:35.869944 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:35.869945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:35.869960 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:35.870003 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:35.870635 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:35.870636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:35.870636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:35.870640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:35.870654 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:35.870658 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:35.870709 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:35.870757 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 23:03:35.870769 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:03:35.873167 29077 raft_consensus.cc:1261] T bc84442a1ec343c68f477c5824436bb4 P 70f1ee41140243d299a682573343f101 [term 1 FOLLOWER]: Refusing update from remote peer a6cdd76859604e969f2372733352e8ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:35.873167 29361 raft_consensus.cc:1261] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [term 1 FOLLOWER]: Refusing update from remote peer a6cdd76859604e969f2372733352e8ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:35.873544 29434 consensus_queue.cc:1035] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "70f1ee41140243d299a682573343f101" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 34335 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:35.873699 29431 consensus_queue.cc:1035] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb391ba4413d4cb89b6ab44f2a506d2b" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40533 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:35.875944 29439 mvcc.cc:205] Tried to move back new op lower bound from 7317106753013010432 to 7317106752828506112. Current Snapshot: MvccSnapshot[applied={T|T < 7317106753013010432}]
I20260810 23:03:35.876212 29441 mvcc.cc:205] Tried to move back new op lower bound from 7317106753013010432 to 7317106752828506112. Current Snapshot: MvccSnapshot[applied={T|T < 7317106753013010432}]
I20260810 23:03:35.877199 29440 mvcc.cc:205] Tried to move back new op lower bound from 7317106753013010432 to 7317106752828506112. Current Snapshot: MvccSnapshot[applied={T|T < 7317106753013010432}]
I20260810 23:03:35.878582 28920 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:35.878674 28920 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:35.879415 28920 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9e3dd344741449eba8679cfefdc0214b: Sending DeleteTablet for 3 replicas of tablet bc84442a1ec343c68f477c5824436bb4
I20260810 23:03:35.879757 29056 tablet_service.cc:1514] Processing DeleteTablet for tablet bc84442a1ec343c68f477c5824436bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:35 UTC) from {username='slave'} at 127.0.0.1:39520
I20260810 23:03:35.879758 29198 tablet_service.cc:1514] Processing DeleteTablet for tablet bc84442a1ec343c68f477c5824436bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:35 UTC) from {username='slave'} at 127.0.0.1:43368
I20260810 23:03:35.880108 29338 tablet_service.cc:1514] Processing DeleteTablet for tablet bc84442a1ec343c68f477c5824436bb4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:35 UTC) from {username='slave'} at 127.0.0.1:53228
I20260810 23:03:35.880144 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 28987
I20260810 23:03:35.881399 29449 tablet_replica.cc:330] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: stopping tablet replica
I20260810 23:03:35.881526 29449 raft_consensus.cc:2229] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:35.881665 29449 raft_consensus.cc:2258] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:35.881816 29451 tablet_replica.cc:330] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: stopping tablet replica
I20260810 23:03:35.881898 29451 raft_consensus.cc:2229] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:35.881978 29451 raft_consensus.cc:2258] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:35.882737 29451 ts_tablet_manager.cc:1905] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:35.882737 29449 ts_tablet_manager.cc:1905] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:35.884613 29451 ts_tablet_manager.cc:1918] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:35.884616 29449 ts_tablet_manager.cc:1918] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:35.884697 29451 log.cc:1196] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/bc84442a1ec343c68f477c5824436bb4
I20260810 23:03:35.884797 29449 log.cc:1196] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/bc84442a1ec343c68f477c5824436bb4
I20260810 23:03:35.884974 29451 ts_tablet_manager.cc:1939] T bc84442a1ec343c68f477c5824436bb4 P fb391ba4413d4cb89b6ab44f2a506d2b: Deleting consensus metadata
I20260810 23:03:35.885210 29449 ts_tablet_manager.cc:1939] T bc84442a1ec343c68f477c5824436bb4 P a6cdd76859604e969f2372733352e8ec: Deleting consensus metadata
I20260810 23:03:35.885366 28905 catalog_manager.cc:4841] TS fb391ba4413d4cb89b6ab44f2a506d2b (127.23.56.67:40533): tablet bc84442a1ec343c68f477c5824436bb4 (table test-table [id=dafa6324dbf0480e97a226a495d2d0a0]) successfully deleted
I20260810 23:03:35.885761 28905 catalog_manager.cc:4841] TS a6cdd76859604e969f2372733352e8ec (127.23.56.66:41219): tablet bc84442a1ec343c68f477c5824436bb4 (table test-table [id=dafa6324dbf0480e97a226a495d2d0a0]) successfully deleted
W20260810 23:03:35.891748 28906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:34335 (error 108)
I20260810 23:03:35.892048 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 29131
I20260810 23:03:35.892271 29420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:35.892149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:34335 (local address 127.0.0.1:39526)
0810 23:03:35.892190 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:35.892234 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:34335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260810 23:03:35.892927 28906 catalog_manager.cc:4584] TS 70f1ee41140243d299a682573343f101 (127.23.56.65:34335): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bc84442a1ec343c68f477c5824436bb4: Network error: Client connection negotiation failed: client connection to 127.23.56.65:34335: connect: Connection refused (error 111)
I20260810 23:03:35.899525 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 29271
I20260810 23:03:35.905444 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 28888
2026-08-10T23:03:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (921 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 23:03:35.926553 23777 mini_oidc.cc:152] Starting JWKS server
I20260810 23:03:35.927764 23777 webserver.cc:460] Webserver started at https://127.0.0.1:40017/ using document root <none> and password file <none>
I20260810 23:03:35.927870 23777 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:03:35.928057 23777 webserver.cc:460] Webserver started at http://127.0.0.1:37455/ using document root <none> and password file <none>
2026-08-10T23:03:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:35Z Disabled control of system clock
I20260810 23:03:35.936724 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:44215
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:44215
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40017/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/testchainca.cert with env {}
W20260810 23:03:36.017243 29459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:36.017442 29459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:36.017463 29459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:36.018558 29459 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:36.018595 29459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:36.018608 29459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:36.018621 29459 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:36.018633 29459 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:03:36.018646 29459 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:03:36.018656 29459 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40017/jwks.json
I20260810 23:03:36.020140 29459 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:44215
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:44215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40017/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:36.020390 29459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:36.020628 29459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:36.025461 29469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:36.025466 29466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:36.025470 29467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:36.025486 29459 server_base.cc:1034] running on GCE node
I20260810 23:03:36.025924 29459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:36.026163 29459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:36.027328 29459 hybrid_clock.cc:648] HybridClock initialized: now 1786403016027304 us; error 36 us; skew 500 ppm
I20260810 23:03:36.028510 29459 webserver.cc:460] Webserver started at http://127.23.56.126:33935/ using document root <none> and password file <none>
I20260810 23:03:36.028738 29459 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:36.028806 29459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:36.028954 29459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:36.030041 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "fe9af3d455574bd8ad27636a9f478bed"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "6f15e08b320d613a10f548667703fb2c"
server_key_iv: "9d9e62d7ea5b78d063dc41e5571ee773"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.030459 29459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "fe9af3d455574bd8ad27636a9f478bed"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "6f15e08b320d613a10f548667703fb2c"
server_key_iv: "9d9e62d7ea5b78d063dc41e5571ee773"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.031917 29459 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:36.032617 29475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:36.032778 29459 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:36.032876 29459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "fe9af3d455574bd8ad27636a9f478bed"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "6f15e08b320d613a10f548667703fb2c"
server_key_iv: "9d9e62d7ea5b78d063dc41e5571ee773"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.032951 29459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:36.043020 29459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:36.043319 29459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:36.043473 29459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:36.047336 29459 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:44215
I20260810 23:03:36.047371 29530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:44215 every 8 connection(s)
I20260810 23:03:36.047700 29459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:36.048722 29532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:36.050765 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 29459
I20260810 23:03:36.050885 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:36.051067 23777 external_mini_cluster.cc:1428] Setting key 453fcaa118274b103adf624c5d29d106
I20260810 23:03:36.051167 29532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed: Bootstrap starting.
I20260810 23:03:36.052024 29532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:36.052367 29532 log.cc:824] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:36.053329 29532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed: No bootstrap required, opened a new log
I20260810 23:03:36.055660 29532 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9af3d455574bd8ad27636a9f478bed" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44215 } }
I20260810 23:03:36.055823 29532 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:36.055857 29532 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe9af3d455574bd8ad27636a9f478bed, State: Initialized, Role: FOLLOWER
I20260810 23:03:36.055994 29532 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9af3d455574bd8ad27636a9f478bed" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44215 } }
I20260810 23:03:36.056079 29532 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:36.056152 29532 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:36.056214 29532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:36.057050 29532 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9af3d455574bd8ad27636a9f478bed" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44215 } }
I20260810 23:03:36.057163 29532 leader_election.cc:302] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe9af3d455574bd8ad27636a9f478bed; no voters:
I20260810 23:03:36.057380 29532 leader_election.cc:288] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:36.057559 29537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:36.057628 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:36.051604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43476 (local address 127.23.56.126:44215)
0810 23:03:36.051826 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:36.051836 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:36.051869 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:03:36.052487 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:36.052498 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:36.052587 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:36.052722 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:36.053106 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.054526 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.055421 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.056019 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.056145 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.056890 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:36.056895 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:36.056895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:36.057105 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:36.057149 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:36.057164 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:36.057235 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:36.057320 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 23:03:36.057402 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":88,"threads_started":1}
I20260810 23:03:36.057670 29532 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:36.057739 29537 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [term 1 LEADER]: Becoming Leader. State: Replica: fe9af3d455574bd8ad27636a9f478bed, State: Running, Role: LEADER
I20260810 23:03:36.057966 29537 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9af3d455574bd8ad27636a9f478bed" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44215 } }
I20260810 23:03:36.058386 29538 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe9af3d455574bd8ad27636a9f478bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9af3d455574bd8ad27636a9f478bed" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44215 } } }
I20260810 23:03:36.058539 29538 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:36.058722 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:03:36.058938 29539 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe9af3d455574bd8ad27636a9f478bed. Latest consensus state: current_term: 1 leader_uuid: "fe9af3d455574bd8ad27636a9f478bed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe9af3d455574bd8ad27636a9f478bed" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44215 } } }
I20260810 23:03:36.059002 29539 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:36.059110 29550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:36.059763 29550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:36.061219 29550 catalog_manager.cc:1347] Generated new cluster ID: 0c7cd564755442db946643dcce4d1a88
I20260810 23:03:36.061260 29550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:36.068455 29550 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:36.068930 29550 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:36.083006 29550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe9af3d455574bd8ad27636a9f478bed: Generated new TSK 0
I20260810 23:03:36.083252 29550 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:03:36.109293 29566 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 23:03:36.109465 29566 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:03:36.109690 29535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:36.093381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43482 (local address 127.23.56.126:44215)
0810 23:03:36.093433 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:36.093436 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:36.093586 (+ 150us) server_negotiation.cc:407] Connection header received
0810 23:03:36.093681 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:36.093682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:36.093695 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:36.093779 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:03:36.093899 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.094702 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.095726 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.096058 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.096131 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.096388 (+ 257us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:03:36.109443 (+ 13055us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:03:36.109612 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":28}
W20260810 23:03:36.109769 29535 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43482: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 23:03:36.110162 29565 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:36.093297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:44215 (local address 127.0.0.1:43482)
0810 23:03:36.093505 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:36.093516 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:36.093580 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:36.093782 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:36.093782 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:36.093784 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:03:36.093835 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:36.093837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.094794 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.094795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:36.095635 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:36.095638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.096163 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.096164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:36.096211 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.096230 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:03:36.109633 (+ 13403us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 23:03:36.109780 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.56.126:44215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.56.126:44215: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40017/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 23:03:36.110682 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 29459
2026-08-10T23:03:36Z chronyd exiting
I20260810 23:03:36.151978 23777 test_util.cc:175] -----------------------------------------------
I20260810 23:03:36.152056 23777 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786402992661514-23777-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 23:03:36.152864 23777 mini_oidc.cc:152] Starting JWKS server
I20260810 23:03:36.153564 23777 webserver.cc:460] Webserver started at https://127.0.0.1:33461/ using document root <none> and password file <none>
I20260810 23:03:36.153609 23777 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:03:36.153795 23777 webserver.cc:460] Webserver started at http://127.0.0.1:35451/ using document root <none> and password file <none>
2026-08-10T23:03:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:36Z Disabled control of system clock
I20260810 23:03:36.162307 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:40489
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:42673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:40489
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33461/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/testchainca.cert with env {}
W20260810 23:03:36.244382 29572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:36.244633 29572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:36.244669 29572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:36.245828 29572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:36.245926 29572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:36.245949 29572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:36.245970 29572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:36.246001 29572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:03:36.246023 29572 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:03:36.246043 29572 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33461/jwks.json
I20260810 23:03:36.247567 29572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:40489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:40489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33461/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:36.247867 29572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:36.248092 29572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:36.253033 29578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:36.252969 29577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:36.252995 29580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:36.253152 29572 server_base.cc:1034] running on GCE node
I20260810 23:03:36.253461 29572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:36.253751 29572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:36.254932 29572 hybrid_clock.cc:648] HybridClock initialized: now 1786403016254905 us; error 37 us; skew 500 ppm
I20260810 23:03:36.256147 29572 webserver.cc:460] Webserver started at http://127.23.56.126:46791/ using document root <none> and password file <none>
I20260810 23:03:36.256399 29572 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:36.256474 29572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:36.256620 29572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:36.257730 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "58fe69acc42042588bd58704902d9542"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "79d6a5a9e439ac73d4676a80fac3d59f"
server_key_iv: "8bf7b7bbdc7a2cd7f052c448f872aebe"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.258196 29572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "58fe69acc42042588bd58704902d9542"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "79d6a5a9e439ac73d4676a80fac3d59f"
server_key_iv: "8bf7b7bbdc7a2cd7f052c448f872aebe"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.259744 29572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:36.260466 29588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:36.260645 29572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:36.260744 29572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "58fe69acc42042588bd58704902d9542"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "79d6a5a9e439ac73d4676a80fac3d59f"
server_key_iv: "8bf7b7bbdc7a2cd7f052c448f872aebe"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.260839 29572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:36.275512 29572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:36.275839 29572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:36.275996 29572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:36.280164 29572 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:40489
I20260810 23:03:36.280206 29647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:40489 every 8 connection(s)
I20260810 23:03:36.280581 29572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:36.281265 29648 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:36.284143 29648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542: Bootstrap starting.
I20260810 23:03:36.284823 29648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:36.285071 29648 log.cc:824] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:36.285728 29648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542: No bootstrap required, opened a new log
I20260810 23:03:36.286566 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 29572
I20260810 23:03:36.286679 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:36.286855 23777 external_mini_cluster.cc:1428] Setting key 53fc8f83ce138659fe4d40aad0e9ffb5
I20260810 23:03:36.287945 29648 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe69acc42042588bd58704902d9542" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40489 } }
I20260810 23:03:36.288120 29648 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:36.288159 29648 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58fe69acc42042588bd58704902d9542, State: Initialized, Role: FOLLOWER
I20260810 23:03:36.288296 29648 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe69acc42042588bd58704902d9542" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40489 } }
I20260810 23:03:36.288386 29648 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:36.288410 29648 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:36.288475 29648 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:36.288997 29648 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe69acc42042588bd58704902d9542" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40489 } }
I20260810 23:03:36.289083 29648 leader_election.cc:302] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58fe69acc42042588bd58704902d9542; no voters:
I20260810 23:03:36.289253 29648 leader_election.cc:288] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:36.289290 29653 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:36.289450 29653 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [term 1 LEADER]: Becoming Leader. State: Replica: 58fe69acc42042588bd58704902d9542, State: Running, Role: LEADER
I20260810 23:03:36.289530 29648 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:36.289585 29653 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe69acc42042588bd58704902d9542" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40489 } }
I20260810 23:03:36.291235 29654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58fe69acc42042588bd58704902d9542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe69acc42042588bd58704902d9542" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40489 } } }
I20260810 23:03:36.291317 29654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:36.291554 29655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58fe69acc42042588bd58704902d9542. Latest consensus state: current_term: 1 leader_uuid: "58fe69acc42042588bd58704902d9542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58fe69acc42042588bd58704902d9542" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40489 } } }
W20260810 23:03:36.291559 29668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 23:03:36.291672 29655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542 [sys.catalog]: This master's current role is: LEADER
W20260810 23:03:36.291754 29668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:03:36.291929 29670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:36.292133 29670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:36.293339 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:36.287527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.23.56.126:40489)
0810 23:03:36.287736 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:36.287743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:36.287767 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:03:36.287903 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:36.287909 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:36.287968 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:36.288085 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:36.288325 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.289861 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.290950 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.291633 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.291762 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.292683 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:36.292687 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:36.292688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:36.292882 (+ 194us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:36.292914 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:36.292926 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:36.293006 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:36.293086 (+ 80us) server_negotiation.cc:299] Negotiation successful
0810 23:03:36.293164 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":94,"threads_started":1}
I20260810 23:03:36.294083 29670 catalog_manager.cc:1347] Generated new cluster ID: 74e022b27f8340a9af965f1cb63f249e
I20260810 23:03:36.294134 29670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:36.294242 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:03:36.304322 29670 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:36.305032 29670 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:36.311058 29670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58fe69acc42042588bd58704902d9542: Generated new TSK 0
I20260810 23:03:36.311272 29670 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 23:03:36.338020 29679 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 23:03:36.338133 29679 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 23:03:36.338424 29652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:36.324235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44992 (local address 127.23.56.126:40489)
0810 23:03:36.324284 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:36.324286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:36.324335 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:03:36.324424 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:36.324425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:36.324435 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:36.324497 (+ 62us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:03:36.324630 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.325105 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.326212 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.326388 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.326434 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.326668 (+ 234us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:03:36.338253 (+ 11585us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:03:36.338374 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 23:03:36.338496 29652 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44992: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 23:03:36.338605 29678 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:36.324096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40489 (local address 127.0.0.1:44992)
0810 23:03:36.324275 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:36.324287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:36.324354 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:36.324505 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:36.324506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:36.324508 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:03:36.324572 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:36.324574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.325175 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.325176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:36.326128 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:36.326131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.326460 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.326461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:36.326521 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.326526 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:03:36.338364 (+ 11838us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 23:03:36.338500 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.56.126:40489: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33461/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":111,"threads_started":1}
I20260810 23:03:36.338922 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 29572
2026-08-10T23:03:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 23:03:36.379238 23777 mini_oidc.cc:152] Starting JWKS server
I20260810 23:03:36.379940 23777 webserver.cc:460] Webserver started at https://127.0.0.1:44185/ using document root <none> and password file <none>
I20260810 23:03:36.380012 23777 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 23:03:36.380170 23777 webserver.cc:460] Webserver started at http://127.0.0.1:46693/ using document root <none> and password file <none>
2026-08-10T23:03:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:36Z Disabled control of system clock
I20260810 23:03:36.389163 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34951
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:33003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:34951
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44185/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:03:36.487911 29685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:36.488128 29685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:36.488150 29685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:36.489259 29685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:36.489296 29685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:36.489310 29685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:36.489322 29685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:36.489336 29685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:03:36.489347 29685 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:03:36.489359 29685 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44185/jwks.json
I20260810 23:03:36.490654 29685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:33003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:34951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44185/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:36.490851 29685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:36.491091 29685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:36.495803 29690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:36.495844 29693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:36.495990 29685 server_base.cc:1034] running on GCE node
W20260810 23:03:36.495862 29691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:36.496342 29685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:36.496587 29685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:36.497745 29685 hybrid_clock.cc:648] HybridClock initialized: now 1786403016497733 us; error 46 us; skew 500 ppm
I20260810 23:03:36.498915 29685 webserver.cc:460] Webserver started at http://127.23.56.126:41935/ using document root <none> and password file <none>
I20260810 23:03:36.499238 29685 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:36.499310 29685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:36.499449 29685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:36.500646 29685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "e566e58620ed43b084c6bee2c425ad20"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "30e143b513ed918fa61a6cfde6ce329a"
server_key_iv: "045f8f93bbed0c11c9583d55e7eb09f4"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.501072 29685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "e566e58620ed43b084c6bee2c425ad20"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "30e143b513ed918fa61a6cfde6ce329a"
server_key_iv: "045f8f93bbed0c11c9583d55e7eb09f4"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.502427 29685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:36.503109 29699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:36.503321 29685 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:36.503386 29685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "e566e58620ed43b084c6bee2c425ad20"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "30e143b513ed918fa61a6cfde6ce329a"
server_key_iv: "045f8f93bbed0c11c9583d55e7eb09f4"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.503436 29685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:36.515424 29685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:36.515679 29685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:36.515777 29685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:36.519577 29685 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:34951
I20260810 23:03:36.519647 29760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:34951 every 8 connection(s)
I20260810 23:03:36.519956 29685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:36.520586 29761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:36.523290 29761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20: Bootstrap starting.
I20260810 23:03:36.523541 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 29685
I20260810 23:03:36.523619 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:36.523804 23777 external_mini_cluster.cc:1428] Setting key 1acb699f39c7bba58c3046d7cce418b0
I20260810 23:03:36.524060 29761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:36.524415 29761 log.cc:824] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:36.525209 29761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20: No bootstrap required, opened a new log
I20260810 23:03:36.527532 29761 raft_consensus.cc:348] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566e58620ed43b084c6bee2c425ad20" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34951 } }
I20260810 23:03:36.527659 29761 raft_consensus.cc:374] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:36.527681 29761 raft_consensus.cc:729] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e566e58620ed43b084c6bee2c425ad20, State: Initialized, Role: FOLLOWER
I20260810 23:03:36.527842 29761 consensus_queue.cc:260] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566e58620ed43b084c6bee2c425ad20" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34951 } }
I20260810 23:03:36.527935 29761 raft_consensus.cc:388] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:36.527971 29761 raft_consensus.cc:482] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:36.528030 29761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:36.528545 29761 raft_consensus.cc:504] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566e58620ed43b084c6bee2c425ad20" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34951 } }
I20260810 23:03:36.528671 29761 leader_election.cc:302] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e566e58620ed43b084c6bee2c425ad20; no voters:
I20260810 23:03:36.528841 29761 leader_election.cc:288] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:36.529032 29769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:36.529110 29761 sys_catalog.cc:564] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:36.529241 29769 raft_consensus.cc:686] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [term 1 LEADER]: Becoming Leader. State: Replica: e566e58620ed43b084c6bee2c425ad20, State: Running, Role: LEADER
I20260810 23:03:36.529346 29769 consensus_queue.cc:237] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566e58620ed43b084c6bee2c425ad20" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34951 } }
I20260810 23:03:36.529697 29769 sys_catalog.cc:455] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e566e58620ed43b084c6bee2c425ad20. Latest consensus state: current_term: 1 leader_uuid: "e566e58620ed43b084c6bee2c425ad20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566e58620ed43b084c6bee2c425ad20" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34951 } } }
I20260810 23:03:36.529781 29769 sys_catalog.cc:458] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:36.529709 29772 sys_catalog.cc:455] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e566e58620ed43b084c6bee2c425ad20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e566e58620ed43b084c6bee2c425ad20" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34951 } } }
I20260810 23:03:36.529881 29772 sys_catalog.cc:458] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:36.530094 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:36.524352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36218 (local address 127.23.56.126:34951)
0810 23:03:36.524570 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:36.524578 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:36.524615 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:03:36.525186 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:36.525194 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:36.525270 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:36.525422 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:36.525601 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.526972 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.527773 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.528271 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.528414 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.529146 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:36.529151 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:36.529151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:36.529375 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:36.529418 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:36.529432 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:36.529474 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:36.529632 (+ 158us) server_negotiation.cc:299] Negotiation successful
0810 23:03:36.529714 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260810 23:03:36.530133 29781 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:36.530572 29781 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:36.531275 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:03:36.532972 29781 catalog_manager.cc:1347] Generated new cluster ID: aa58f5d973a4450fb65fcbfafc116c52
I20260810 23:03:36.533031 29781 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:36.550092 29781 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:36.550591 29781 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:36.558326 29781 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e566e58620ed43b084c6bee2c425ad20: Generated new TSK 0
I20260810 23:03:36.558513 29781 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:03:36.568266 29766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:36.564809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.23.56.126:34951)
0810 23:03:36.564882 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:36.564885 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:36.564985 (+ 100us) server_negotiation.cc:407] Connection header received
0810 23:03:36.565099 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:36.565100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:36.565118 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:36.565196 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:03:36.565338 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.565839 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.566648 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.566862 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.566939 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.567165 (+ 226us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:03:36.568059 (+ 894us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44185/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:03:36.568191 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36232: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44185/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 23:03:36.568320 29766 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36232: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44185/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 23:03:36.568372 29796 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:36.564733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34951 (local address 127.0.0.1:36232)
0810 23:03:36.564904 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:36.564916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:36.565011 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:36.565195 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:36.565197 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:36.565199 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:03:36.565257 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:36.565259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.565924 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.565926 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:36.566556 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:36.566558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.566945 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.566946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:36.567015 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.567022 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:03:36.568164 (+ 1142us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44185/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 23:03:36.568270 (+ 106us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.56.126:34951: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44185/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260810 23:03:36.568535 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 29685
E20260810 23:03:36.582623 29798 webserver.cc:533] Webserver: sslize failed:
E20260810 23:03:36.582696 29798 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T23:03:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T23:03:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:36Z Disabled control of system clock
I20260810 23:03:36.620656 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:44135
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:44543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:44135
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/testchainca.cert with env {}
W20260810 23:03:36.701858 29802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:36.702062 29802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:36.702085 29802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:36.703289 29802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:36.703330 29802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:36.703343 29802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:36.703374 29802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:36.703405 29802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 23:03:36.703426 29802 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 23:03:36.703451 29802 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 23:03:36.704998 29802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:44543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:44135
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:44135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:36.705236 29802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:36.705454 29802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:36.710261 29808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:36.710268 29811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:36.710281 29809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:36.710500 29802 server_base.cc:1034] running on GCE node
I20260810 23:03:36.710748 29802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:36.711031 29802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:36.712190 29802 hybrid_clock.cc:648] HybridClock initialized: now 1786403016712161 us; error 39 us; skew 500 ppm
I20260810 23:03:36.713341 29802 webserver.cc:460] Webserver started at http://127.23.56.126:39161/ using document root <none> and password file <none>
I20260810 23:03:36.713577 29802 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:36.713654 29802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:36.713804 29802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:36.714879 29802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "7a67ba20f553497ab254bfca12b0bdfe"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "f932ac557409cda2de1f9040cc160f4f"
server_key_iv: "1f7e4062b7f80ba7d9e669e7373a8281"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.715310 29802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "7a67ba20f553497ab254bfca12b0bdfe"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "f932ac557409cda2de1f9040cc160f4f"
server_key_iv: "1f7e4062b7f80ba7d9e669e7373a8281"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.716729 29802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:36.717415 29817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:36.717584 29802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:36.717686 29802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "7a67ba20f553497ab254bfca12b0bdfe"
format_stamp: "Formatted at 2026-08-10 23:03:36 on dist-test-slave-1xrh"
server_key: "f932ac557409cda2de1f9040cc160f4f"
server_key_iv: "1f7e4062b7f80ba7d9e669e7373a8281"
server_key_version: "encryptionkey@0"
I20260810 23:03:36.717772 29802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:36.729341 29802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:36.729640 29802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:36.729785 29802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:36.733649 29802 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:44135
I20260810 23:03:36.733683 29876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:44135 every 8 connection(s)
I20260810 23:03:36.734023 29802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:36.734913 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 29802
I20260810 23:03:36.735029 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:36.735143 29877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:36.735263 23777 external_mini_cluster.cc:1428] Setting key d318867f5e23e788f435ba6ae63c2565
I20260810 23:03:36.738155 29877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe: Bootstrap starting.
I20260810 23:03:36.738847 29877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:36.739253 29877 log.cc:824] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:36.739976 29877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe: No bootstrap required, opened a new log
I20260810 23:03:36.740935 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:36.735790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45614 (local address 127.23.56.126:44135)
0810 23:03:36.735989 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:36.735997 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:36.736021 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:03:36.736555 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:36.736562 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:36.736618 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:36.736728 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:36.736961 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.737778 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.738851 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.739412 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.739551 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.740243 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:36.740247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:36.740247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:03:36.740437 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:36.740489 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:03:36.740504 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:36.740574 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:36.740632 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 23:03:36.740710 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260810 23:03:36.742169 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:03:36.742480 29877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a67ba20f553497ab254bfca12b0bdfe" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44135 } }
I20260810 23:03:36.742607 29877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:36.742628 29877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a67ba20f553497ab254bfca12b0bdfe, State: Initialized, Role: FOLLOWER
I20260810 23:03:36.742760 29877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a67ba20f553497ab254bfca12b0bdfe" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44135 } }
I20260810 23:03:36.742858 29877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:36.742897 29877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:36.742946 29877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:36.743594 29877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a67ba20f553497ab254bfca12b0bdfe" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44135 } }
I20260810 23:03:36.743710 29877 leader_election.cc:302] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a67ba20f553497ab254bfca12b0bdfe; no voters:
I20260810 23:03:36.743873 29877 leader_election.cc:288] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:36.743918 29883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:36.744079 29883 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [term 1 LEADER]: Becoming Leader. State: Replica: 7a67ba20f553497ab254bfca12b0bdfe, State: Running, Role: LEADER
I20260810 23:03:36.744144 29877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:36.744163 29883 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a67ba20f553497ab254bfca12b0bdfe" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44135 } }
I20260810 23:03:36.744536 29885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a67ba20f553497ab254bfca12b0bdfe. Latest consensus state: current_term: 1 leader_uuid: "7a67ba20f553497ab254bfca12b0bdfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a67ba20f553497ab254bfca12b0bdfe" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44135 } } }
I20260810 23:03:36.744629 29885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:36.744529 29884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a67ba20f553497ab254bfca12b0bdfe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a67ba20f553497ab254bfca12b0bdfe" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44135 } } }
I20260810 23:03:36.744796 29884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:36.745177 29890 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:36.745685 29890 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:36.747206 29890 catalog_manager.cc:1347] Generated new cluster ID: cffcee44de654b0094011fd362e724cd
I20260810 23:03:36.747248 29890 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:36.756347 29890 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:36.756829 29890 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:36.762070 29890 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a67ba20f553497ab254bfca12b0bdfe: Generated new TSK 0
I20260810 23:03:36.762233 29890 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:03:37.267894 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:36.773203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.23.56.126:44135)
0810 23:03:36.773258 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:36.773260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:36.773445 (+ 185us) server_negotiation.cc:407] Connection header received
0810 23:03:36.773497 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:36.773498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:36.773512 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:36.773587 (+ 75us) server_negotiation.cc:226] Negotiated authn=JWT
0810 23:03:36.773748 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.774230 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.775389 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.775578 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.775629 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.775816 (+ 187us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 23:03:37.267657 (+491841us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:03:37.267826 (+ 169us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45620: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 23:03:37.268006 29881 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45620: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 23:03:37.268016 29908 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:36.773134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:44135 (local address 127.0.0.1:45620)
0810 23:03:36.773294 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:36.773305 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:36.773365 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:36.773615 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:36.773616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:36.773619 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0810 23:03:36.773682 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:36.773684 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.774316 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.774317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:36.775313 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:36.775316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:36.775656 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:36.775657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:36.775715 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:36.775721 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 23:03:37.267808 (+492087us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 23:03:37.267939 (+ 131us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.56.126:44135: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260810 23:03:37.268397 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 29802
2026-08-10T23:03:37Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (676 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (95 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (90 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:37 dist-test-slave-1xrh krb5kdc[29923](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:37 dist-test-slave-1xrh krb5kdc[29923](info): set up 2 sockets
Aug 10 23:03:37 dist-test-slave-1xrh krb5kdc[29923](info): commencing operation
krb5kdc: starting...
W20260810 23:03:38.620234 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.145s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:38 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:38.743643 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34045
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:34045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:38.826815 29936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:38.827049 29936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:38.827090 29936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:38.828279 29936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:38.828349 29936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:38.828377 29936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:38.828408 29936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:38.828434 29936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:38.829887 29936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:34045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:38.830169 29936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:38.830387 29936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:38.835289 29942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:38.835259 29943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:38.835294 29936 server_base.cc:1034] running on GCE node
W20260810 23:03:38.835260 29945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:38.835786 29936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:38.836025 29936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:38.837167 29936 hybrid_clock.cc:648] HybridClock initialized: now 1786403018837138 us; error 42 us; skew 500 ppm
Aug 10 23:03:38 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403018, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:38.839778 29936 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:38.840193 29936 webserver.cc:460] Webserver started at http://127.23.56.126:33303/ using document root <none> and password file <none>
I20260810 23:03:38.840427 29936 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:38.840471 29936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:38.840624 29936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:38.841638 29936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "53967a9e9ed447248c479ae2d1fb5cd2"
format_stamp: "Formatted at 2026-08-10 23:03:38 on dist-test-slave-1xrh"
server_key: "508dfd5f74a6b4c54092a0d91315b284"
server_key_iv: "90571ffdc5010dbe52db2f8b1ea9b468"
server_key_version: "encryptionkey@0"
I20260810 23:03:38.841985 29936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "53967a9e9ed447248c479ae2d1fb5cd2"
format_stamp: "Formatted at 2026-08-10 23:03:38 on dist-test-slave-1xrh"
server_key: "508dfd5f74a6b4c54092a0d91315b284"
server_key_iv: "90571ffdc5010dbe52db2f8b1ea9b468"
server_key_version: "encryptionkey@0"
I20260810 23:03:38.843308 29936 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:03:38.844017 29952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:38.844204 29936 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:38.844292 29936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "53967a9e9ed447248c479ae2d1fb5cd2"
format_stamp: "Formatted at 2026-08-10 23:03:38 on dist-test-slave-1xrh"
server_key: "508dfd5f74a6b4c54092a0d91315b284"
server_key_iv: "90571ffdc5010dbe52db2f8b1ea9b468"
server_key_version: "encryptionkey@0"
I20260810 23:03:38.844364 29936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:38.869194 29936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:38.871068 29936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:38.871249 29936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:38.875166 29936 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:34045
I20260810 23:03:38.875243 30007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:34045 every 8 connection(s)
I20260810 23:03:38.875597 29936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:38.876477 30009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:38.878468 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 29936
I20260810 23:03:38.878585 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:38.878782 23777 external_mini_cluster.cc:1428] Setting key 7aa7d7755e8c9eef6ab88af3393f98ae
I20260810 23:03:38.878971 30009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2: Bootstrap starting.
I20260810 23:03:38.879703 30009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:38.880016 30009 log.cc:824] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:38.880759 30009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2: No bootstrap required, opened a new log
I20260810 23:03:38.882544 30009 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53967a9e9ed447248c479ae2d1fb5cd2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34045 } }
I20260810 23:03:38.882699 30009 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:38.882748 30009 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53967a9e9ed447248c479ae2d1fb5cd2, State: Initialized, Role: FOLLOWER
I20260810 23:03:38.882882 30009 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53967a9e9ed447248c479ae2d1fb5cd2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34045 } }
I20260810 23:03:38.882942 30009 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:38.882962 30009 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:38.883028 30009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:38.883571 30009 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53967a9e9ed447248c479ae2d1fb5cd2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34045 } }
I20260810 23:03:38.883953 30009 leader_election.cc:302] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53967a9e9ed447248c479ae2d1fb5cd2; no voters:
I20260810 23:03:38.884172 30009 leader_election.cc:288] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:38.884197 30016 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:38.884322 30016 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [term 1 LEADER]: Becoming Leader. State: Replica: 53967a9e9ed447248c479ae2d1fb5cd2, State: Running, Role: LEADER
I20260810 23:03:38.884436 30009 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:38.884461 30016 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53967a9e9ed447248c479ae2d1fb5cd2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34045 } }
I20260810 23:03:38.884814 30016 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53967a9e9ed447248c479ae2d1fb5cd2. Latest consensus state: current_term: 1 leader_uuid: "53967a9e9ed447248c479ae2d1fb5cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53967a9e9ed447248c479ae2d1fb5cd2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34045 } } }
I20260810 23:03:38.884827 30017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53967a9e9ed447248c479ae2d1fb5cd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53967a9e9ed447248c479ae2d1fb5cd2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34045 } } }
I20260810 23:03:38.884943 30016 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:38.885216 30017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:38.885385 30026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 23:03:38 dist-test-slave-1xrh krb5kdc[29923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:38.885977 30026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:38.889309 30026 catalog_manager.cc:1347] Generated new cluster ID: e7082c2811ae4d53bc8fabb57c62f3ba
I20260810 23:03:38.889359 30026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:38.889732 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:38.879348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46936 (local address 127.23.56.126:34045)
0810 23:03:38.879555 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:38.879565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:38.879601 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:03:38.880093 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:38.880103 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:38.880165 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:38.880337 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:38.881065 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:38.881930 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:38.882707 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:38.883308 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:38.883431 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:38.886330 (+ 2899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:38.886335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:38.886335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:38.886347 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:38.888042 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:38.888437 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:38.888440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:38.888441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:38.888531 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:38.888752 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:38.888753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:38.888753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:38.889077 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:38.889247 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:38.889375 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 23:03:38.889519 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":91,"threads_started":1}
I20260810 23:03:38.893249 30026 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:38.893733 30026 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:38.902571 30026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53967a9e9ed447248c479ae2d1fb5cd2: Generated new TSK 0
I20260810 23:03:38.902748 30026 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:38.932365 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34045
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:39.014945 30038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:39.015156 30038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:39.015179 30038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:39.016319 30038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:39.016359 30038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:39.016410 30038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:39.017915 30038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:39.018157 30038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:39.018388 30038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:39.023344 30044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:39.023342 30045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:39.023342 30047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:39.023550 30038 server_base.cc:1034] running on GCE node
I20260810 23:03:39.023804 30038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:39.024063 30038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:39.025229 30038 hybrid_clock.cc:648] HybridClock initialized: now 1786403019025222 us; error 34 us; skew 500 ppm
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.027809 30038 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:39.028272 30038 webserver.cc:460] Webserver started at http://127.23.56.65:45859/ using document root <none> and password file <none>
I20260810 23:03:39.028497 30038 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:39.028565 30038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:39.028715 30038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:39.029712 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "0182c9c31c834eb6a27f235f59f06613"
format_stamp: "Formatted at 2026-08-10 23:03:39 on dist-test-slave-1xrh"
server_key: "97e53a8d734015f42f5b247ca8b440b7"
server_key_iv: "ff9dc756c378350292ba9876b50c0d38"
server_key_version: "encryptionkey@0"
I20260810 23:03:39.030068 30038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "0182c9c31c834eb6a27f235f59f06613"
format_stamp: "Formatted at 2026-08-10 23:03:39 on dist-test-slave-1xrh"
server_key: "97e53a8d734015f42f5b247ca8b440b7"
server_key_iv: "ff9dc756c378350292ba9876b50c0d38"
server_key_version: "encryptionkey@0"
I20260810 23:03:39.031394 30038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:03:39.032083 30058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:39.032238 30038 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:39.032339 30038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "0182c9c31c834eb6a27f235f59f06613"
format_stamp: "Formatted at 2026-08-10 23:03:39 on dist-test-slave-1xrh"
server_key: "97e53a8d734015f42f5b247ca8b440b7"
server_key_iv: "ff9dc756c378350292ba9876b50c0d38"
server_key_version: "encryptionkey@0"
I20260810 23:03:39.032413 30038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:39.051887 30038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:39.053718 30038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:39.053887 30038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:39.054136 30038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:39.054461 30038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:39.054529 30038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:39.054569 30038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:39.054598 30038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:39.060436 30038 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:44245
I20260810 23:03:39.060489 30180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:44245 every 8 connection(s)
I20260810 23:03:39.060806 30038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:39.067584 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 30038
I20260810 23:03:39.067713 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:39.067945 23777 external_mini_cluster.cc:1428] Setting key bdcf10a7596a3fde05710e56829e6a9d
I20260810 23:03:39.071326 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:39.061574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:47581 (local address 127.23.56.126:34045)
0810 23:03:39.061653 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:39.061656 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:39.061862 (+ 206us) server_negotiation.cc:407] Connection header received
0810 23:03:39.062879 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:39.062881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:39.062893 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:39.062950 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:39.063635 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.064170 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.065186 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.065407 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.065496 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:39.067909 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:39.067911 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:39.067911 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:39.067920 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:39.069570 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.070098 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.070099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.070100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.070152 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.070459 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.070460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.070460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.070628 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:39.070706 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:39.071060 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 23:03:39.071195 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 23:03:39.071434 30184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:39.061474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34045 (local address 127.23.56.65:47581)
0810 23:03:39.061706 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:39.061760 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:39.062723 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:39.063019 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:39.063024 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:39.063327 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:39.063521 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:39.063526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.064273 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.064274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:39.065087 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:39.065089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.065507 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.065508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:39.065637 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:39.066140 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:39.066149 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:39.067806 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:39.069658 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:39.069666 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:39.069668 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:39.070010 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:39.070216 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:39.070217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:39.070217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:39.070375 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:39.070790 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:39.070797 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:39.070917 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 23:03:39.071033 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 23:03:39.071238 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":90,"threads_started":1}
I20260810 23:03:39.072156 30181 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34045
I20260810 23:03:39.072268 30181 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:39.072448 30181 heartbeater.cc:511] Master 127.23.56.126:34045 requested a full tablet report, sending...
I20260810 23:03:39.072988 29971 ts_manager.cc:194] Registered new tserver with Master: 0182c9c31c834eb6a27f235f59f06613 (127.23.56.65:44245)
I20260810 23:03:39.073688 29971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:47581
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:39.097909 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34045
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:39.179875 30187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:39.180076 30187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:39.180099 30187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:39.181334 30187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:39.181377 30187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:39.181404 30187 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:39.182843 30187 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:39.183096 30187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:39.183363 30187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:39.188452 30193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:39.188401 30192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:39.188621 30187 server_base.cc:1034] running on GCE node
W20260810 23:03:39.188401 30195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:39.188922 30187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:39.189167 30187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:39.190326 30187 hybrid_clock.cc:648] HybridClock initialized: now 1786403019190313 us; error 26 us; skew 500 ppm
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.193035 30187 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:39.193404 30187 webserver.cc:460] Webserver started at http://127.23.56.66:44019/ using document root <none> and password file <none>
I20260810 23:03:39.193579 30187 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:39.193619 30187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:39.193701 30187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:39.194654 30187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "9ff1d74f39b94f5ebc417e36fd2fecda"
format_stamp: "Formatted at 2026-08-10 23:03:39 on dist-test-slave-1xrh"
server_key: "aa129131156fd3153b25bdfa1129b5e5"
server_key_iv: "4894ae734ae02d92664f134a796fa1cc"
server_key_version: "encryptionkey@0"
I20260810 23:03:39.194942 30187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "9ff1d74f39b94f5ebc417e36fd2fecda"
format_stamp: "Formatted at 2026-08-10 23:03:39 on dist-test-slave-1xrh"
server_key: "aa129131156fd3153b25bdfa1129b5e5"
server_key_iv: "4894ae734ae02d92664f134a796fa1cc"
server_key_version: "encryptionkey@0"
I20260810 23:03:39.196233 30187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:39.196874 30203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:39.197039 30187 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:39.197139 30187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "9ff1d74f39b94f5ebc417e36fd2fecda"
format_stamp: "Formatted at 2026-08-10 23:03:39 on dist-test-slave-1xrh"
server_key: "aa129131156fd3153b25bdfa1129b5e5"
server_key_iv: "4894ae734ae02d92664f134a796fa1cc"
server_key_version: "encryptionkey@0"
I20260810 23:03:39.197212 30187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:39.219981 30187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:39.221740 30187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:39.221902 30187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:39.222138 30187 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:39.222445 30187 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:39.222505 30187 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:39.222568 30187 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:39.222599 30187 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:39.228603 30187 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:38411
I20260810 23:03:39.228641 30323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:38411 every 8 connection(s)
I20260810 23:03:39.228968 30187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:39.233263 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 30187
I20260810 23:03:39.233371 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:39.233577 23777 external_mini_cluster.cc:1428] Setting key 8038bb1b3f45f93f110f97d03b039fcf
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:39.238742 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:39.229717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:50135 (local address 127.23.56.126:34045)
0810 23:03:39.229796 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:39.229798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:39.229974 (+ 176us) server_negotiation.cc:407] Connection header received
0810 23:03:39.230853 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:39.230855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:39.230865 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:39.230916 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:39.231600 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.232146 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.233129 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.233447 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.233530 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:39.235957 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:39.235959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:39.235959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:39.235967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:39.237327 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.237741 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.237743 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.237743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.237811 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.238075 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.238076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.238076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.238196 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:39.238272 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:39.238560 (+ 288us) server_negotiation.cc:299] Negotiation successful
0810 23:03:39.238654 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:03:39.238895 30326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:39.229670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34045 (local address 127.23.56.66:50135)
0810 23:03:39.229849 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:39.229884 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:39.230766 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:39.230968 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:39.230971 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:39.231307 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:39.231494 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:39.231498 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.232233 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.232233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:39.233037 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:39.233039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.233539 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.233540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:39.233640 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:39.234183 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:39.234206 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:39.235871 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:39.237402 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:39.237408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:39.237409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:39.237659 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:39.237880 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:39.237881 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:39.237881 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:39.237996 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:39.238275 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:39.238282 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:39.238414 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 23:03:39.238521 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:03:39.238672 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260810 23:03:39.239403 30324 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34045
I20260810 23:03:39.239496 30324 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:39.239691 30324 heartbeater.cc:511] Master 127.23.56.126:34045 requested a full tablet report, sending...
I20260810 23:03:39.240104 29971 ts_manager.cc:194] Registered new tserver with Master: 9ff1d74f39b94f5ebc417e36fd2fecda (127.23.56.66:38411)
I20260810 23:03:39.240657 29971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:50135
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:39.262166 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34045
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:39.347154 30329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:39.347371 30329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:39.347396 30329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:39.348596 30329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:39.348646 30329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:39.348673 30329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:39.350080 30329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:39.350260 30329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:39.350445 30329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:39.355600 30335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:39.355600 30336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:39.355643 30338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:39.355626 30329 server_base.cc:1034] running on GCE node
I20260810 23:03:39.355930 30329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:39.356201 30329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:39.357337 30329 hybrid_clock.cc:648] HybridClock initialized: now 1786403019357326 us; error 29 us; skew 500 ppm
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.359877 30329 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:39.360350 30329 webserver.cc:460] Webserver started at http://127.23.56.67:34121/ using document root <none> and password file <none>
I20260810 23:03:39.360610 30329 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:39.360656 30329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:39.360816 30329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:39.361853 30329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "3b77bd503bf24be2ba75659151a841d0"
format_stamp: "Formatted at 2026-08-10 23:03:39 on dist-test-slave-1xrh"
server_key: "02e722fb8d308470bef3844c713da4cf"
server_key_iv: "65207f9f755b4d81658515e4d9427063"
server_key_version: "encryptionkey@0"
I20260810 23:03:39.362210 30329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "3b77bd503bf24be2ba75659151a841d0"
format_stamp: "Formatted at 2026-08-10 23:03:39 on dist-test-slave-1xrh"
server_key: "02e722fb8d308470bef3844c713da4cf"
server_key_iv: "65207f9f755b4d81658515e4d9427063"
server_key_version: "encryptionkey@0"
I20260810 23:03:39.363670 30329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 23:03:39.364403 30346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:39.364586 30329 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:39.364651 30329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "3b77bd503bf24be2ba75659151a841d0"
format_stamp: "Formatted at 2026-08-10 23:03:39 on dist-test-slave-1xrh"
server_key: "02e722fb8d308470bef3844c713da4cf"
server_key_iv: "65207f9f755b4d81658515e4d9427063"
server_key_version: "encryptionkey@0"
I20260810 23:03:39.364741 30329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:39.389716 30329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:39.391666 30329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:39.391775 30329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:39.391966 30329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:39.392211 30329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:39.392241 30329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:39.392262 30329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:39.392275 30329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:39.398211 30329 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:40179
I20260810 23:03:39.398243 30481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:40179 every 8 connection(s)
I20260810 23:03:39.398582 30329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:39.407461 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 30329
I20260810 23:03:39.407574 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:39.407763 23777 external_mini_cluster.cc:1428] Setting key 28cd08d1a71aae5a94d9ae665b178ee5
I20260810 23:03:39.408314 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:39.399399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:53905 (local address 127.23.56.126:34045)
0810 23:03:39.399449 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:39.399452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:39.399663 (+ 211us) server_negotiation.cc:407] Connection header received
0810 23:03:39.400735 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:39.400736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:39.400749 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:39.400830 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:39.401454 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.401999 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.403268 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.403475 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.403565 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:39.405617 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:39.405619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:39.405619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:39.405626 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:39.406788 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.407196 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.407197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.407197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.407248 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.407565 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.407566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.407566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.407714 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:39.407766 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:39.408133 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 23:03:39.408213 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:03:39.408496 30484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:39.399356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34045 (local address 127.23.56.67:53905)
0810 23:03:39.399542 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:39.399578 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:39.400624 (+ 1046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:39.400902 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:39.400906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:39.401175 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:39.401359 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:39.401363 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.402124 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.402125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:39.403174 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:39.403180 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.403576 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.403577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:39.403702 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:39.404165 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:39.404172 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:39.405533 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:39.406855 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:39.406860 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:39.406862 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:39.407080 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:39.407336 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:39.407337 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:39.407337 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:39.407475 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:39.407780 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:39.407787 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:39.407949 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 23:03:39.408086 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 23:03:39.408255 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260810 23:03:39.408919 30482 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34045
I20260810 23:03:39.409025 30482 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:39.409199 30482 heartbeater.cc:511] Master 127.23.56.126:34045 requested a full tablet report, sending...
I20260810 23:03:39.409639 29971 ts_manager.cc:194] Registered new tserver with Master: 3b77bd503bf24be2ba75659151a841d0 (127.23.56.67:40179)
I20260810 23:03:39.410050 29971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:53905
I20260810 23:03:39.419770 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/test-admin.keytab.
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.436280 23777 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 23:03:39.437834 30490 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:39.437130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.23.56.65:44245)
0810 23:03:39.437292 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:39.437296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:39.437306 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:39.437330 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:39.437332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:39.437345 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:39.437430 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:39.437772 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47392: BlockingRecv error: recv got EOF from 127.0.0.1:47392 (error 108)
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
W20260810 23:03:39.438007 23777 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.438967 23777 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.439915 23777 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:39.447185 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:39.440396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47406 (local address 127.23.56.65:44245)
0810 23:03:39.440448 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:39.440450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:39.440506 (+ 56us) server_negotiation.cc:407] Connection header received
0810 23:03:39.440550 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:39.440551 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:39.440562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:39.440608 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:39.440969 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.441667 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.442347 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.442649 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.442700 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:39.444696 (+ 1996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:39.444703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:39.444703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:39.444710 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:39.446056 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.446395 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.446399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.446399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.446449 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.446606 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.446607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.446607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.446718 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:39.446848 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:39.446941 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 23:03:39.447031 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260810 23:03:39.448935 30490 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:39.448418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.23.56.65:44245)
0810 23:03:39.448467 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:39.448468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:39.448478 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:39.448569 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:39.448569 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:39.448578 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:39.448633 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:39.448867 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47414: BlockingRecv error: recv got EOF from 127.0.0.1:47414 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260810 23:03:39.449039 23777 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.449892 23777 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.450671 23777 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:39.456996 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:39.451216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47420 (local address 127.23.56.65:44245)
0810 23:03:39.451263 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:39.451265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:39.451314 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:03:39.451351 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:39.451351 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:39.451360 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:39.451407 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:39.451741 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.452199 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.452821 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.453017 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.453064 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:39.454923 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:39.454924 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:39.454924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:39.454930 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:39.456144 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.456433 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.456433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.456434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.456480 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.456639 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.456640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.456640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.456721 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:39.456787 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:39.456876 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:03:39.456931 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 23:03:39.458631 30490 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:39.457953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47428 (local address 127.23.56.65:44245)
0810 23:03:39.458012 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:39.458013 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:39.458058 (+ 45us) server_negotiation.cc:407] Connection header received
0810 23:03:39.458127 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:39.458128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:39.458137 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:39.458209 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:39.458592 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47428: BlockingRecv error: recv got EOF from 127.0.0.1:47428 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260810 23:03:39.458843 23777 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786402992661514-23777-0/krb5kdc/krb5cc' of type FILE
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.459888 23777 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:39.460748 23777 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[29923](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403019, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:03:39.467020 30490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:39.461275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47438 (local address 127.23.56.65:44245)
0810 23:03:39.461309 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:39.461310 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:39.461352 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:03:39.461399 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:39.461400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:39.461407 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:39.461453 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:39.461792 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.462247 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.462866 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:39.463040 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:39.463086 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:39.465081 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:39.465083 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:39.465083 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:39.465089 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:39.466190 (+ 1101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.466473 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.466473 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.466474 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.466521 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:39.466676 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:39.466676 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:39.466677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:39.466756 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:39.466819 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:39.466905 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 23:03:39.466955 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:03:39.467877 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 30038
I20260810 23:03:39.475658 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 30187
I20260810 23:03:39.484645 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 30329
I20260810 23:03:39.490311 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 29936
2026-08-10T23:03:39Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2036 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[30504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[30504](info): set up 2 sockets
Aug 10 23:03:39 dist-test-slave-1xrh krb5kdc[30504](info): commencing operation
krb5kdc: starting...
W20260810 23:03:40.601605 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:40 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:40Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.126.keytab.
Entry for principal oryx/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.126.keytab.
I20260810 23:03:40.722144 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:40607
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.126.keytab
--principal=oryx/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:40607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:40.807932 30519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:40.808182 30519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:40.808223 30519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:40.809388 30519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:40.809453 30519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:40.809489 30519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:40.809517 30519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:40.809541 30519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:40.811008 30519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:40607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.126.keytab
--principal=oryx/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:40.811303 30519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:40.811538 30519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:40.816226 30525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:40.816242 30524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:40.816473 30519 server_base.cc:1034] running on GCE node
W20260810 23:03:40.816234 30527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:40.816792 30519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:40.817117 30519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:40.818259 30519 hybrid_clock.cc:648] HybridClock initialized: now 1786403020818245 us; error 28 us; skew 500 ppm
Aug 10 23:03:40 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403020, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:40.820688 30519 init.cc:377] Logged in from keytab as oryx/127.23.56.126@KRBTEST.COM (short username oryx)
I20260810 23:03:40.821118 30519 webserver.cc:460] Webserver started at http://127.23.56.126:33517/ using document root <none> and password file <none>
I20260810 23:03:40.821300 30519 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:40.821341 30519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:40.821429 30519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:40.822403 30519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "307f9e72b6a3437ebf590125864218af"
format_stamp: "Formatted at 2026-08-10 23:03:40 on dist-test-slave-1xrh"
server_key: "bcf8835f152a6e4c2311acad14e3164b"
server_key_iv: "711253e40943b03c39e1772be1da291e"
server_key_version: "encryptionkey@0"
I20260810 23:03:40.822710 30519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "307f9e72b6a3437ebf590125864218af"
format_stamp: "Formatted at 2026-08-10 23:03:40 on dist-test-slave-1xrh"
server_key: "bcf8835f152a6e4c2311acad14e3164b"
server_key_iv: "711253e40943b03c39e1772be1da291e"
server_key_version: "encryptionkey@0"
I20260810 23:03:40.824044 30519 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:40.824723 30534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:40.824910 30519 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:40.825004 30519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "307f9e72b6a3437ebf590125864218af"
format_stamp: "Formatted at 2026-08-10 23:03:40 on dist-test-slave-1xrh"
server_key: "bcf8835f152a6e4c2311acad14e3164b"
server_key_iv: "711253e40943b03c39e1772be1da291e"
server_key_version: "encryptionkey@0"
I20260810 23:03:40.825058 30519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:40.838333 30519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:40.840130 30519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:40.840237 30519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:40.844009 30519 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:40607
I20260810 23:03:40.844060 30594 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:40607 every 8 connection(s)
I20260810 23:03:40.844383 30519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:40.845347 30595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:40.847010 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 30519
I20260810 23:03:40.847182 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:40.847386 23777 external_mini_cluster.cc:1428] Setting key 96d2a9753f004466093b86873ec93c61
I20260810 23:03:40.848168 30595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: Bootstrap starting.
I20260810 23:03:40.848825 30595 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:40.849160 30595 log.cc:824] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:40.849895 30595 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: No bootstrap required, opened a new log
I20260810 23:03:40.852072 30595 raft_consensus.cc:348] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } }
I20260810 23:03:40.852246 30595 raft_consensus.cc:374] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:40.852288 30595 raft_consensus.cc:729] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 307f9e72b6a3437ebf590125864218af, State: Initialized, Role: FOLLOWER
I20260810 23:03:40.852447 30595 consensus_queue.cc:260] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } }
I20260810 23:03:40.852558 30595 raft_consensus.cc:388] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:40.852593 30595 raft_consensus.cc:482] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:40.852640 30595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:40.853166 30595 raft_consensus.cc:504] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } }
I20260810 23:03:40.853262 30595 leader_election.cc:302] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 307f9e72b6a3437ebf590125864218af; no voters:
I20260810 23:03:40.853461 30595 leader_election.cc:288] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:40.853482 30600 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:40.853609 30600 raft_consensus.cc:686] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 1 LEADER]: Becoming Leader. State: Replica: 307f9e72b6a3437ebf590125864218af, State: Running, Role: LEADER
I20260810 23:03:40.853686 30600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } }
I20260810 23:03:40.853757 30595 sys_catalog.cc:564] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:40.853987 30600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } }
I20260810 23:03:40.854043 30600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:40.854168 30600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307f9e72b6a3437ebf590125864218af. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } }
I20260810 23:03:40.854223 30600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
Aug 10 23:03:40 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:40.854346 30604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:40.855063 30604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:40.856655 30604 catalog_manager.cc:1347] Generated new cluster ID: f51bfab748804bfb9d9da94266c4c310
I20260810 23:03:40.856705 30604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:40.858906 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:40.847937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47754 (local address 127.23.56.126:40607)
0810 23:03:40.848115 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:40.848123 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:40.848148 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:03:40.848604 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:40.848611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:40.848659 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:40.848799 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:40.849605 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:40.850549 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:40.851641 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:40.852020 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:40.852098 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:40.854914 (+ 2816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:40.854918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:40.854919 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:40.854931 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:40.857163 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:40.857557 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:40.857560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:40.857561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:40.857674 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:40.857904 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:40.857905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:40.857905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:40.858249 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:40.858394 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:40.858501 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 23:03:40.858671 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260810 23:03:40.871161 30604 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:40.871682 30604 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:40.878402 30604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: Generated new TSK 0
I20260810 23:03:40.878592 30604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab.
Entry for principal oryx/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab.
I20260810 23:03:40.913941 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:40607
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab
--principal=oryx/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:40.997876 30622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:40.998090 30622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:40.998114 30622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:40.999334 30622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:40.999377 30622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:40.999408 30622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:41.000838 30622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab
--principal=oryx/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:40607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:41.001032 30622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:41.001216 30622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:41.006145 30630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.006153 30627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.006158 30628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:41.006455 30622 server_base.cc:1034] running on GCE node
I20260810 23:03:41.006676 30622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:41.006868 30622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:41.008005 30622 hybrid_clock.cc:648] HybridClock initialized: now 1786403021007993 us; error 34 us; skew 500 ppm
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:41.010637 30622 init.cc:377] Logged in from keytab as oryx/127.23.56.65@KRBTEST.COM (short username oryx)
I20260810 23:03:41.011193 30622 webserver.cc:460] Webserver started at http://127.23.56.65:37367/ using document root <none> and password file <none>
I20260810 23:03:41.011400 30622 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:41.011441 30622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:41.011538 30622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:41.012543 30622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "12146b8872474a2ab2c251e990776763"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "1bd76ffced16f07bf7ff710e76003dda"
server_key_iv: "306a88ff55b8b8c4b946912e76c20e41"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.012903 30622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "12146b8872474a2ab2c251e990776763"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "1bd76ffced16f07bf7ff710e76003dda"
server_key_iv: "306a88ff55b8b8c4b946912e76c20e41"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.014207 30622 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:41.014928 30638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.015111 30622 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:41.015239 30622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "12146b8872474a2ab2c251e990776763"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "1bd76ffced16f07bf7ff710e76003dda"
server_key_iv: "306a88ff55b8b8c4b946912e76c20e41"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.015323 30622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:41.025271 30622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:41.027057 30622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:41.027276 30622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:41.027514 30622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:41.027839 30622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:41.027895 30622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.027937 30622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:41.027963 30622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.033710 30622 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:43395
I20260810 23:03:41.033792 30768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:43395 every 8 connection(s)
I20260810 23:03:41.034114 30622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:41.039536 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 30622
I20260810 23:03:41.039650 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:41.039850 23777 external_mini_cluster.cc:1428] Setting key 31fd45d6c73cda51ddd55b245c2a17f0
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:41.043838 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.034913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:45963 (local address 127.23.56.126:40607)
0810 23:03:41.034979 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.034981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.035167 (+ 186us) server_negotiation.cc:407] Connection header received
0810 23:03:41.035681 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.035682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.035692 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.035746 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.036674 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.037182 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.038118 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.038311 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.038361 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.040676 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.040677 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.040677 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.040685 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.042314 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.042828 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.042829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.042829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.042880 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.043178 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.043178 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.043179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.043288 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.043341 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.043680 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.043760 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:03:41.044047 30771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.034840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.65:45963)
0810 23:03:41.035022 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.035066 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.035595 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.035805 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.035809 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.036399 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.036597 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.036602 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.037262 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.037263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.038045 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.038047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.038385 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.038386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.038490 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.039038 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.039046 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.040609 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.042412 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.042421 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.042423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.042751 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.042949 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.042950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.042950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.043090 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.043354 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.043361 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.043530 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.043648 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.043830 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
I20260810 23:03:41.044641 30769 heartbeater.cc:347] Connected to a master server at 127.23.56.126:40607
I20260810 23:03:41.044729 30769 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:41.044926 30769 heartbeater.cc:511] Master 127.23.56.126:40607 requested a full tablet report, sending...
I20260810 23:03:41.045507 30553 ts_manager.cc:194] Registered new tserver with Master: 12146b8872474a2ab2c251e990776763 (127.23.56.65:43395)
I20260810 23:03:41.046447 30553 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:45963
WARNING: no policy specified for oryx/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab.
Entry for principal oryx/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab.
I20260810 23:03:41.068138 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:40607
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab
--principal=oryx/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:41.151584 30774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:41.151783 30774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:41.151806 30774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:41.152879 30774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:41.152911 30774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:41.152936 30774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:41.154488 30774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab
--principal=oryx/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:40607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:41.154683 30774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:41.154860 30774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:41.159802 30780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.159790 30779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.159791 30782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:41.160063 30774 server_base.cc:1034] running on GCE node
I20260810 23:03:41.160281 30774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:41.160519 30774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:41.161684 30774 hybrid_clock.cc:648] HybridClock initialized: now 1786403021161670 us; error 27 us; skew 500 ppm
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:41.164304 30774 init.cc:377] Logged in from keytab as oryx/127.23.56.66@KRBTEST.COM (short username oryx)
I20260810 23:03:41.164726 30774 webserver.cc:460] Webserver started at http://127.23.56.66:35919/ using document root <none> and password file <none>
I20260810 23:03:41.164953 30774 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:41.165027 30774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:41.165182 30774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:41.166268 30774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "9e2e115077a142aba3475eccdcbd1890"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "ccadc71f8f83e6b435d445c6bcb008db"
server_key_iv: "48ed06a7e71e419afe561c8a64dbb0c7"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.166636 30774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "9e2e115077a142aba3475eccdcbd1890"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "ccadc71f8f83e6b435d445c6bcb008db"
server_key_iv: "48ed06a7e71e419afe561c8a64dbb0c7"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.168033 30774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:03:41.168725 30791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.168959 30774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:41.169075 30774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "9e2e115077a142aba3475eccdcbd1890"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "ccadc71f8f83e6b435d445c6bcb008db"
server_key_iv: "48ed06a7e71e419afe561c8a64dbb0c7"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.169153 30774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:41.178727 30774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:41.181084 30774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:41.181252 30774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:41.181465 30774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:41.181813 30774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:41.181879 30774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.181921 30774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:41.181949 30774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.188019 30774 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:42249
I20260810 23:03:41.188062 30912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:42249 every 8 connection(s)
I20260810 23:03:41.188392 30774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:41.193459 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 30774
I20260810 23:03:41.193584 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:41.193773 23777 external_mini_cluster.cc:1428] Setting key e687ed35a5a9cc9e1ffe6fec969a22f1
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:41.198138 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.189136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:44443 (local address 127.23.56.126:40607)
0810 23:03:41.189207 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.189209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.189377 (+ 168us) server_negotiation.cc:407] Connection header received
0810 23:03:41.189858 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.189859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.189869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.189929 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.190884 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.191470 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.192463 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.192652 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.192702 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.195027 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.195029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.195029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.195038 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.196614 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.197109 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.197110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.197110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.197160 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.197444 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.197445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.197445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.197541 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.197588 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.197915 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.198001 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:03:41.198153 30918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.189075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.66:44443)
0810 23:03:41.189238 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.189281 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.189763 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.189981 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.189985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.190582 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.190782 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.190793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.191584 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.191585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.192382 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.192384 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.192733 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.192735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.192870 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.193442 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.193450 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.194928 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.196693 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.196701 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.196703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.197032 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.197228 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.197228 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.197229 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.197372 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.197603 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.197610 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.197768 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.197858 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.197977 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260810 23:03:41.198793 30913 heartbeater.cc:347] Connected to a master server at 127.23.56.126:40607
I20260810 23:03:41.198909 30913 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:41.199165 30913 heartbeater.cc:511] Master 127.23.56.126:40607 requested a full tablet report, sending...
I20260810 23:03:41.199581 30553 ts_manager.cc:194] Registered new tserver with Master: 9e2e115077a142aba3475eccdcbd1890 (127.23.56.66:42249)
I20260810 23:03:41.200098 30553 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:44443
WARNING: no policy specified for oryx/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab.
Entry for principal oryx/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab.
I20260810 23:03:41.224020 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:40607
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab
--principal=oryx/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:41.313263 30922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:41.313570 30922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:41.313609 30922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:41.315424 30922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:41.315500 30922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:41.315541 30922 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:41.317890 30922 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab
--principal=oryx/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:40607
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:41.318183 30922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:41.318495 30922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:41.324663 30929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.324584 30931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.324604 30928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:41.324756 30922 server_base.cc:1034] running on GCE node
I20260810 23:03:41.325040 30922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:41.325309 30922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:41.326500 30922 hybrid_clock.cc:648] HybridClock initialized: now 1786403021326479 us; error 44 us; skew 500 ppm
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:41.329632 30922 init.cc:377] Logged in from keytab as oryx/127.23.56.67@KRBTEST.COM (short username oryx)
I20260810 23:03:41.330207 30922 webserver.cc:460] Webserver started at http://127.23.56.67:41725/ using document root <none> and password file <none>
I20260810 23:03:41.330482 30922 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:41.330556 30922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:41.330711 30922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:41.331949 30922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "7530190c7d1445818b024acf973c940f"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "61ccaedd6b80cd61a82c25dc71159f6e"
server_key_iv: "4c0b96467f46a1c51b7a27284f78b79c"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.332345 30922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "7530190c7d1445818b024acf973c940f"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "61ccaedd6b80cd61a82c25dc71159f6e"
server_key_iv: "4c0b96467f46a1c51b7a27284f78b79c"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.333737 30922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:03:41.334618 30938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.334968 30922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:41.335210 30922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "7530190c7d1445818b024acf973c940f"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "61ccaedd6b80cd61a82c25dc71159f6e"
server_key_iv: "4c0b96467f46a1c51b7a27284f78b79c"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.335335 30922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:41.343999 30922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:41.346381 30922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:41.346547 30922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:41.346791 30922 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:41.347240 30922 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:41.347294 30922 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.347325 30922 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:41.347347 30922 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.354099 30922 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:35465
I20260810 23:03:41.354142 31067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:35465 every 8 connection(s)
I20260810 23:03:41.354519 30922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:41.361183 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 30922
I20260810 23:03:41.361342 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:41.361824 23777 external_mini_cluster.cc:1428] Setting key 4be684f741aae74b82060ff65b3fb544
I20260810 23:03:41.365115 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.355416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:43551 (local address 127.23.56.126:40607)
0810 23:03:41.355536 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.355554 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.355774 (+ 220us) server_negotiation.cc:407] Connection header received
0810 23:03:41.356396 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.356399 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.356414 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.356461 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.357453 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.357992 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.359067 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.359370 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.359448 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.361811 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.361813 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.361814 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.361825 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.363550 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.364071 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.364073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.364073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.364141 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.364397 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.364399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.364399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.364524 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.364599 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.364937 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.365025 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 23:03:41.365288 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.355423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.67:43551)
0810 23:03:41.355632 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.355683 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.356264 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.356542 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.356546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.357138 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.357346 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.357351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.358097 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.358098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.358954 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.358957 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.359457 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.359458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.359598 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.360108 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.360117 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.361680 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.363664 (+ 1984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.363671 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.363673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.363978 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.364206 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.364207 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.364208 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.364326 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.364589 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.364595 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.364747 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.364873 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.365047 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":116,"threads_started":1}
I20260810 23:03:41.365759 31068 heartbeater.cc:347] Connected to a master server at 127.23.56.126:40607
I20260810 23:03:41.365845 31068 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:41.366045 31068 heartbeater.cc:511] Master 127.23.56.126:40607 requested a full tablet report, sending...
I20260810 23:03:41.366480 30553 ts_manager.cc:194] Registered new tserver with Master: 7530190c7d1445818b024acf973c940f (127.23.56.67:35465)
I20260810 23:03:41.366897 30553 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:43551
I20260810 23:03:41.374539 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.56.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.125.keytab.
Entry for principal oryx/127.23.56.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.125.keytab.
I20260810 23:03:41.404367 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.125:45849
--webserver_interface=127.23.56.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.125.keytab
--principal=oryx/127.23.56.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:40607,127.23.56.125:45849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:41.487767 31073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:41.488009 31073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:41.488044 31073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:41.489212 31073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:41.489275 31073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:41.489300 31073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:41.489341 31073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:41.489367 31073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:41.490893 31073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:40607,127.23.56.125:45849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.125.keytab
--principal=oryx/127.23.56.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.125:45849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.56.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
W20260810 23:03:41.491047 31073 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.56.126:40607,127.23.56.125:45849'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 23:03:41.491236 31073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:41.491492 31073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:41.496366 31081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.496367 31080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.496389 31083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:41.496625 31073 server_base.cc:1034] running on GCE node
I20260810 23:03:41.496887 31073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:41.497171 31073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:41.498339 31073 hybrid_clock.cc:648] HybridClock initialized: now 1786403021498317 us; error 39 us; skew 500 ppm
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:41.501057 31073 init.cc:377] Logged in from keytab as oryx/127.23.56.125@KRBTEST.COM (short username oryx)
I20260810 23:03:41.501516 31073 webserver.cc:460] Webserver started at http://127.23.56.125:39835/ using document root <none> and password file <none>
I20260810 23:03:41.501773 31073 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:41.501889 31073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:41.502043 31073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:41.503166 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data/instance:
uuid: "90454ac770814be18a301f882e97b0a0"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "e1987e326299eb46b34834b5b1fbf09b"
server_key_iv: "33cf76cd5b4e4475dbf9ea8d14bdb02c"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.503549 31073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal/instance:
uuid: "90454ac770814be18a301f882e97b0a0"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "e1987e326299eb46b34834b5b1fbf09b"
server_key_iv: "33cf76cd5b4e4475dbf9ea8d14bdb02c"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.504962 31073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:41.505757 31091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.506031 31073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:41.506166 31073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal
uuid: "90454ac770814be18a301f882e97b0a0"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "e1987e326299eb46b34834b5b1fbf09b"
server_key_iv: "33cf76cd5b4e4475dbf9ea8d14bdb02c"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.506244 31073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:41.517483 31073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:41.519935 31073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:41.520093 31073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:41.524120 31073 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.125:45849
I20260810 23:03:41.524156 31150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.125:45849 every 8 connection(s)
I20260810 23:03:41.524530 31073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
I20260810 23:03:41.525521 31151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:41.527019 31151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:41.529728 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 31073
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.125@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.56.125@KRBTEST.COM
I20260810 23:03:41.537763 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.527769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47756 (local address 127.23.56.126:40607)
0810 23:03:41.527834 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.527836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.528065 (+ 229us) server_negotiation.cc:407] Connection header received
0810 23:03:41.528577 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.528578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.528590 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.528658 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.529697 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.530575 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.531913 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.532114 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.532168 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.534736 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.534738 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.534739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.534747 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.536215 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.536672 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.536673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.536674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.536742 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.537061 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.537062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.537062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.537231 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.537315 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.537576 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.537663 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:03:41.537995 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.527737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:47756)
0810 23:03:41.527911 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.527945 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.528476 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.528825 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.528830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.529264 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.529607 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.529612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.530699 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.530700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.531827 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.531829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.532195 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.532196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.532330 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.532854 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.532862 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.534644 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.536287 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.536292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.536294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.536579 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.536805 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.536806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.536807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.536911 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.537305 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.537311 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.537439 (+ 128us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.537553 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.537729 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
I20260810 23:03:41.538730 31151 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:03:41.539180 31155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.530270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40134 (local address 127.23.56.125:45849)
0810 23:03:41.530440 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.530443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.530458 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:03:41.530519 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.530522 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.530541 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.530616 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.531277 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.532253 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.533110 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.533566 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.533622 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.536164 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.536172 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.536172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.536179 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.537802 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.538221 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.538224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.538225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.538287 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.538545 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.538547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.538547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.538714 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.538868 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.538965 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.539046 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":92,"threads_started":1}
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.125@KRBTEST.COM for oryx/127.23.56.125@KRBTEST.COM
I20260810 23:03:41.545418 31153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.538980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.0.0.1:40146)
0810 23:03:41.539197 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.539207 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.539375 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.539536 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.539536 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.539667 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.539707 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.539708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.540401 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.540401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.540867 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.540868 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.541260 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.541261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.541324 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.541636 (+ 312us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.541640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.542684 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.544629 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.544630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.544631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.544787 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.544952 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.544953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.544953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.545004 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.545211 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.545211 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.545244 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.545292 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.545363 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260810 23:03:41.545588 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.539032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40146 (local address 127.23.56.125:45849)
0810 23:03:41.539201 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.539203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.539273 (+ 70us) server_negotiation.cc:407] Connection header received
0810 23:03:41.539455 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.539456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.539467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.539525 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.539786 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.540313 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.540971 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.541191 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.541263 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.542753 (+ 1490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.542754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.542754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.542759 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.544560 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.544850 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.544851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.544851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.544897 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.545072 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.545072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.545072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.545158 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.545205 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.545305 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.545405 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260810 23:03:41.546866 31151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Bootstrap starting.
I20260810 23:03:41.547580 31151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:41.547866 31151 log.cc:824] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:41.548578 31151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: No bootstrap required, opened a new log
I20260810 23:03:41.549976 31151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } }
I20260810 23:03:41.550101 31151 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:41.550123 31151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90454ac770814be18a301f882e97b0a0, State: Initialized, Role: FOLLOWER
I20260810 23:03:41.550275 31151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } }
I20260810 23:03:41.550567 31151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:41.550529 31159 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } } }
I20260810 23:03:41.550638 31159 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:03:41.552292 31172 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:03:41.552346 31172 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:03:41.554191 31073 master_runner.cc:428] Detected that this master 90454ac770814be18a301f882e97b0a0 is joining an existing cluster
I20260810 23:03:41.554231 31073 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.56.125:45849
I20260810 23:03:41.562304 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.555301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.0.0.1:40152)
0810 23:03:41.555498 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.555509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.555578 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.555742 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.555742 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.556008 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.556066 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.556068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.556968 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.556969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.557820 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.557823 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.558238 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.558239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.558292 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.558748 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.558752 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.559185 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.561178 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.561179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.561179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.561396 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.561618 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.561619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.561619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.561675 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.561926 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.561927 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.561976 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.562065 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.562171 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":134,"threads_started":1}
I20260810 23:03:41.562372 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.555311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40152 (local address 127.23.56.125:45849)
0810 23:03:41.555492 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.555494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.555585 (+ 91us) server_negotiation.cc:407] Connection header received
0810 23:03:41.555635 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.555636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.555646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.555704 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.556161 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.556888 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.557909 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.558139 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.558222 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.559273 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.559275 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.559275 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.559282 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.561106 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.561493 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.561495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.561495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.561547 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.561744 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.561745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.561746 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.561852 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.561921 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.562087 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.562252 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":88,"server-negotiator.queue_time_us":166}
I20260810 23:03:41.563380 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.555359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47772 (local address 127.23.56.126:40607)
0810 23:03:41.555401 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.555404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.555597 (+ 193us) server_negotiation.cc:407] Connection header received
0810 23:03:41.555674 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.555676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.555686 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.555743 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.556118 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.556890 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.558881 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.559215 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.559275 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.560407 (+ 1132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.560409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.560409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.560416 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.562030 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.562654 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.562656 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.562656 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.562714 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.562898 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.562899 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.562899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.563014 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.563069 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.563258 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.563319 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:03:41.563546 31180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.555315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:47772)
0810 23:03:41.555541 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.555549 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.555619 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.555748 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.555749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.555968 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.556039 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.556041 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.557929 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.557929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.558791 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.558794 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.559326 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.559327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.559383 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.559919 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.559924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.560308 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.562124 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.562125 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.562125 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.562565 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.562785 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.562785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.562786 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.562843 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.563089 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.563090 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.563160 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.563230 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.563325 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":136,"threads_started":1}
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403020, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.126@KRBTEST.COM for oryx/127.23.56.125@KRBTEST.COM
I20260810 23:03:41.572778 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.565877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40164 (local address 127.23.56.125:45849)
0810 23:03:41.565911 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.565913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.566075 (+ 162us) server_negotiation.cc:407] Connection header received
0810 23:03:41.566125 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.566126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.566136 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.566196 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.566549 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.567258 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.567967 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.568256 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.568323 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.570164 (+ 1841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.570165 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.570166 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.570173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.571606 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.572025 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.572026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.572027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.572077 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.572277 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.572278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.572278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.572364 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.572411 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.572601 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.572660 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:03:41.572808 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.565810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.0.0.1:40164)
0810 23:03:41.565943 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.566013 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.566074 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.566205 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.566210 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.566436 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.566479 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.566480 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.567326 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.567326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.567891 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.567893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.568329 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.568330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.568381 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.568791 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.568795 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.570085 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.571699 (+ 1614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.571706 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.571708 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.571951 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.572146 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.572147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.572147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.572216 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.572425 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.572432 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.572482 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.572577 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.572692 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":84,"threads_started":1}
I20260810 23:03:41.573704 30553 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.56.125:45849: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "90454ac770814be18a301f882e97b0a0"
member_type: NON_VOTER
last_known_addr {
host: "127.23.56.125"
port: 45849
}
attrs {
promote: true
}
}
dest_uuid: "307f9e72b6a3437ebf590125864218af"
cas_config_opid_index: -1
I20260810 23:03:41.574011 30553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } }
I20260810 23:03:41.574548 31183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 90454ac770814be18a301f882e97b0a0 (127.23.56.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } } }
I20260810 23:03:41.574795 31121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:41.575614 31183 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.56.125:45849
I20260810 23:03:41.575631 31121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Refusing update from remote peer 307f9e72b6a3437ebf590125864218af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 23:03:41.575932 31073 master.cc:537] Master@127.23.56.125:45849 shutting down...
I20260810 23:03:41.575940 31183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Connected to new peer: Peer: permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:41.576579 31182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } } }
W20260810 23:03:41.576646 30537 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260810 23:03:41.576678 31182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
W20260810 23:03:41.577070 30537 consensus_peers.cc:487] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af -> Peer 90454ac770814be18a301f882e97b0a0 (127.23.56.125:45849): Couldn't send request to peer 90454ac770814be18a301f882e97b0a0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:03:41.578644 31073 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:41.578729 31073 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:41.578780 31073 tablet_replica.cc:330] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: stopping tablet replica
I20260810 23:03:41.591527 31073 master.cc:552] Master@127.23.56.125:45849 shutdown complete.
I20260810 23:03:41.591593 31073 master_runner.cc:305] Clearing existing system tablet
I20260810 23:03:41.591724 31073 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:41.591784 31073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:41.592424 31073 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:41.592921 31190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.593144 31073 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:41.593221 31073 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:41.593250 31073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal
uuid: "90454ac770814be18a301f882e97b0a0"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "e1987e326299eb46b34834b5b1fbf09b"
server_key_iv: "33cf76cd5b4e4475dbf9ea8d14bdb02c"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.594131 31073 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:41.595710 31073 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:03:41.595785 31073 log.cc:1196] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 23:03:41.595995 31073 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Deleting consensus metadata
I20260810 23:03:41.596165 31073 master_runner.cc:315] Copying system tablet from 127.23.56.126:40607
I20260810 23:03:41.596786 31073 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: tablet copy: Beginning tablet copy session from remote peer at address 127.23.56.126:40607
I20260810 23:03:41.603741 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.597361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.23.56.126:40607)
0810 23:03:41.597429 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.597432 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.597577 (+ 145us) server_negotiation.cc:407] Connection header received
0810 23:03:41.597638 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.597639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.597650 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.597702 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.598076 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.598576 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.599708 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.599925 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.599979 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.601015 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.601016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.601016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.601023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.602523 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.602904 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.602905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.602905 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.602959 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.603236 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.603237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.603238 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.603367 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.603424 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.603615 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.603677 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:03:41.603825 31196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.597254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:47788)
0810 23:03:41.597484 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.597504 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.597574 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.597725 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.597726 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.597926 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.597993 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.597995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.598678 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.598679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.599613 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.599617 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.599999 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.600000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.600068 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.600530 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.600536 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.600928 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.602605 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.602606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.602607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.602828 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.603027 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.603028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.603029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.603097 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.603436 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.603437 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.603501 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.603584 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.603697 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":126,"threads_started":1}
I20260810 23:03:41.604110 30577 tablet_copy_service.cc:140] P 307f9e72b6a3437ebf590125864218af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 90454ac770814be18a301f882e97b0a0 ({username='oryx', principal='oryx/127.23.56.125@KRBTEST.COM'} at 127.0.0.1:47788)
I20260810 23:03:41.604203 30577 tablet_copy_service.cc:161] P 307f9e72b6a3437ebf590125864218af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 90454ac770814be18a301f882e97b0a0 at {username='oryx', principal='oryx/127.23.56.125@KRBTEST.COM'} at 127.0.0.1:47788: session id = 90454ac770814be18a301f882e97b0a0-00000000000000000000000000000000
I20260810 23:03:41.604828 30577 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:03:41.605553 31073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:03:41.606791 31073 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: tablet copy: Starting download of 0 data blocks...
I20260810 23:03:41.606940 31073 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: tablet copy: Starting download of 1 WAL segments...
I20260810 23:03:41.607957 31073 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:03:41.608891 30577 tablet_copy_service.cc:342] P 307f9e72b6a3437ebf590125864218af: Request end of tablet copy session 90454ac770814be18a301f882e97b0a0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.56.125@KRBTEST.COM'} at 127.0.0.1:47788
I20260810 23:03:41.608982 30577 tablet_copy_service.cc:433] P 307f9e72b6a3437ebf590125864218af: ending tablet copy session 90454ac770814be18a301f882e97b0a0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 90454ac770814be18a301f882e97b0a0
I20260810 23:03:41.611967 31073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:41.613219 31197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.613284 31198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:41.613453 31073 server_base.cc:1034] running on GCE node
W20260810 23:03:41.613520 31200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:41.613686 31073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:41.613898 31073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:41.615025 31073 hybrid_clock.cc:648] HybridClock initialized: now 1786403021615014 us; error 28 us; skew 500 ppm
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:41.616420 31073 init.cc:377] Logged in from keytab as oryx/127.23.56.125@KRBTEST.COM (short username oryx)
I20260810 23:03:41.616719 31073 webserver.cc:460] Webserver started at http://127.23.56.125:33477/ using document root <none> and password file <none>
I20260810 23:03:41.616866 31073 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:41.616906 31073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:41.617550 31073 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:41.618175 31207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.618322 31073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:41.618368 31073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal
uuid: "90454ac770814be18a301f882e97b0a0"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "e1987e326299eb46b34834b5b1fbf09b"
server_key_iv: "33cf76cd5b4e4475dbf9ea8d14bdb02c"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.618427 31073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:41.642596 31073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:41.644287 31073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:41.647701 31073 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.125:45849
I20260810 23:03:41.647720 31267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.125:45849 every 8 connection(s)
I20260810 23:03:41.648046 31073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
I20260810 23:03:41.648983 31268 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:03:41.649544 31268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Bootstrap starting.
I20260810 23:03:41.652540 31268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:03:41.652782 31268 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Bootstrap complete.
I20260810 23:03:41.652976 31268 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } }
I20260810 23:03:41.653064 31268 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 90454ac770814be18a301f882e97b0a0, State: Initialized, Role: LEARNER
I20260810 23:03:41.653102 31268 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } }
I20260810 23:03:41.653278 31268 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:41.653286 31271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } } }
I20260810 23:03:41.653518 31271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:03:41.654951 31282 catalog_manager.cc:1259] Loaded cluster ID: f51bfab748804bfb9d9da94266c4c310
I20260810 23:03:41.655097 31282 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: loading cluster ID for follower catalog manager: success
I20260810 23:03:41.656062 31282 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: acquiring CA information for follower catalog manager: success
I20260810 23:03:41.656272 31282 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:03:41.698414 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.691430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40204 (local address 127.23.56.125:45849)
0810 23:03:41.691576 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.691577 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.691587 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:41.691612 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.691613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.691623 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.691693 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.692371 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.693102 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.694136 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.694319 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.694370 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.695797 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.695798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.695799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.695805 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.697285 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.697660 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.697660 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.697661 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.697709 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.697892 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.697893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.697893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.698015 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.698089 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.698211 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.698278 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
WARNING: no policy specified for oryx/127.23.56.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.124.keytab.
Entry for principal oryx/127.23.56.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.124.keytab.
I20260810 23:03:41.727505 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.124:45181
--webserver_interface=127.23.56.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.124.keytab
--principal=oryx/127.23.56.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:40607,127.23.56.125:45849,127.23.56.124:45181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:41.812933 31287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:41.813140 31287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:41.813162 31287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:41.814311 31287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:41.814354 31287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:41.814368 31287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:41.814380 31287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:41.814393 31287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:41.815974 31287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:40607,127.23.56.125:45849,127.23.56.124:45181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.124.keytab
--principal=oryx/127.23.56.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.124:45181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.56.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:41.816262 31287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:41.816496 31287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:41.821578 31294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.821548 31296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.821556 31293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:41.821862 31287 server_base.cc:1034] running on GCE node
I20260810 23:03:41.822065 31287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:41.822373 31287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:41.823547 31287 hybrid_clock.cc:648] HybridClock initialized: now 1786403021823534 us; error 38 us; skew 500 ppm
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:41.826083 31287 init.cc:377] Logged in from keytab as oryx/127.23.56.124@KRBTEST.COM (short username oryx)
I20260810 23:03:41.826470 31287 webserver.cc:460] Webserver started at http://127.23.56.124:41979/ using document root <none> and password file <none>
I20260810 23:03:41.826663 31287 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:41.826700 31287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:41.826788 31287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:41.828002 31287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data/instance:
uuid: "5e1ca015590a4be6ab5503c5e9b24f74"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "d71fc43fb3d0f92a17be9f326ba04646"
server_key_iv: "0d85c946e7b4c144a6eab33b3f7b5534"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.828310 31287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal/instance:
uuid: "5e1ca015590a4be6ab5503c5e9b24f74"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "d71fc43fb3d0f92a17be9f326ba04646"
server_key_iv: "0d85c946e7b4c144a6eab33b3f7b5534"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.829574 31287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:41.830245 31303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.830415 31287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:41.830492 31287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal
uuid: "5e1ca015590a4be6ab5503c5e9b24f74"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "d71fc43fb3d0f92a17be9f326ba04646"
server_key_iv: "0d85c946e7b4c144a6eab33b3f7b5534"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.830545 31287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:41.843050 31287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:41.845418 31287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:41.845587 31287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:41.850050 31287 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.124:45181
I20260810 23:03:41.850124 31364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.124:45181 every 8 connection(s)
I20260810 23:03:41.850441 31287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
I20260810 23:03:41.851490 31365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:41.852885 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 31287
I20260810 23:03:41.853130 31365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.124@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403020, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.56.124@KRBTEST.COM
I20260810 23:03:41.862546 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.854032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47804 (local address 127.23.56.126:40607)
0810 23:03:41.854079 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.854081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.854182 (+ 101us) server_negotiation.cc:407] Connection header received
0810 23:03:41.854634 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.854635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.854647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.854695 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.855575 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.856066 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.856959 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.857168 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.857243 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.859524 (+ 2281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.859525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.859526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.859532 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.861139 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.861559 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.861561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.861561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.861622 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.861849 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.861850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.861851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.862001 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.862069 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.862379 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.862465 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 23:03:41.862758 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.853878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:47804)
0810 23:03:41.854057 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.854101 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.854526 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.854764 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.854769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.855234 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.855475 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.855488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.856168 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.856169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.856856 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.856858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.857269 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.857271 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.857374 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.857943 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.857954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.859437 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.861224 (+ 1787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.861230 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.861232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.861478 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.861694 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.861694 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.861695 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.861783 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.862074 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.862080 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.862233 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.862345 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.862510 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":64,"threads_started":1}
I20260810 23:03:41.863370 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.853520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49274 (local address 127.23.56.124:45181)
0810 23:03:41.853773 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.853783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.853820 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:03:41.854516 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.854524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.854547 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.854660 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.855438 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.856226 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.856958 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.857261 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.857377 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.860168 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.860176 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.860176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.860183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.861934 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.862347 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.862351 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.862351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.862411 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.862650 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.862651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.862651 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.862800 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.862986 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.863081 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.863259 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":117,"threads_started":1}
I20260810 23:03:41.863343 31365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.124@KRBTEST.COM for oryx/127.23.56.125@KRBTEST.COM
I20260810 23:03:41.869943 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.863607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.0.0.1:40206)
0810 23:03:41.863655 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.863666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.863769 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.863947 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.863948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.864088 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.864122 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.864123 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.864765 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.864766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.865264 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.865265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.865743 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.865744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.865807 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.866126 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.866130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.867337 (+ 1207us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.868953 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.868954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.868954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.869111 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.869301 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.869301 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.869301 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.869372 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.869724 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.869725 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.869760 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.869822 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.869888 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 23:03:41.869992 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.863708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.23.56.125:45849)
0810 23:03:41.863747 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.863749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.863771 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:03:41.863849 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.863850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.863863 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.863929 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.864204 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.864683 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.865357 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.865669 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.865746 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.867431 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.867432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.867433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.867440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.868873 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.869177 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.869177 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.869178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.869226 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.869513 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.869514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.869515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.869635 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.869703 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.869839 (+ 136us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.869910 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:03:41.870488 31365 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.124@KRBTEST.COM for oryx/127.23.56.124@KRBTEST.COM
I20260810 23:03:41.876274 31368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.870741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.0.0.1:49280)
0810 23:03:41.870790 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.870800 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.870836 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.870947 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.870947 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.871053 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.871077 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.871078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.871755 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.871756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.872210 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.872211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.872579 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.872580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.872615 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.872907 (+ 292us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.872910 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.873952 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.875542 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.875543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.875543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.875687 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.875840 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.875841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.875841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.875886 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.876072 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.876073 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.876103 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.876152 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.876220 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 23:03:41.876335 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.870789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49280 (local address 127.23.56.124:45181)
0810 23:03:41.870818 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.870820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.870827 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:03:41.870893 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.870894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.870901 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.870945 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.871115 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.871689 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.872278 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.872518 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.872565 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.874021 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.874023 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.874023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.874029 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.875475 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.875743 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.875744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.875744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.875788 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.875922 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.875922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.875922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.876021 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.876066 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.876167 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.876264 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":15}
I20260810 23:03:41.877606 31365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: Bootstrap starting.
I20260810 23:03:41.878424 31365 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:41.878787 31365 log.cc:824] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:41.879567 31365 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: No bootstrap required, opened a new log
I20260810 23:03:41.880977 31365 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } }
I20260810 23:03:41.881107 31365 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:41.881155 31365 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e1ca015590a4be6ab5503c5e9b24f74, State: Initialized, Role: FOLLOWER
I20260810 23:03:41.881299 31365 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } }
I20260810 23:03:41.881610 31365 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:41.881582 31370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } } }
I20260810 23:03:41.881687 31370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:03:41.883306 31384 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:03:41.883361 31384 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:03:41.885653 31287 master_runner.cc:428] Detected that this master 5e1ca015590a4be6ab5503c5e9b24f74 is joining an existing cluster
I20260810 23:03:41.885700 31287 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.56.124:45181
I20260810 23:03:41.909427 31370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:41.909614 31370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } }
I20260810 23:03:41.909914 31370 leader_election.cc:288] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 307f9e72b6a3437ebf590125864218af (127.23.56.126:40607), 90454ac770814be18a301f882e97b0a0 (127.23.56.125:45849)
I20260810 23:03:41.910192 31238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90454ac770814be18a301f882e97b0a0" is_pre_election: true
I20260810 23:03:41.910324 31238 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 LEARNER]: Handling vote request from an unknown peer 5e1ca015590a4be6ab5503c5e9b24f74
I20260810 23:03:41.910245 30565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "307f9e72b6a3437ebf590125864218af" is_pre_election: true
I20260810 23:03:41.910379 31238 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 5e1ca015590a4be6ab5503c5e9b24f74 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260810 23:03:41.910645 31308 leader_election.cc:302] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1ca015590a4be6ab5503c5e9b24f74; no voters: 307f9e72b6a3437ebf590125864218af, 90454ac770814be18a301f882e97b0a0
I20260810 23:03:41.910772 31370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:41.911630 31370 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 23:03:41.914667 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 30622
I20260810 23:03:41.924254 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:43395
--local_ip_for_outbound_sockets=127.23.56.65
--tserver_master_addrs=127.23.56.126:40607
--webserver_port=37367
--webserver_interface=127.23.56.65
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab
--principal=oryx/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.56.126:40607,127.23.56.125:45849
--tserver_master_addrs=127.23.56.126:40607,127.23.56.125:45849,127.23.56.124:45181 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:03:41.924844 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.887185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47816 (local address 127.23.56.126:40607)
0810 23:03:41.887227 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.887229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.887488 (+ 259us) server_negotiation.cc:407] Connection header received
0810 23:03:41.887584 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.887586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.887596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.887625 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.888000 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.888718 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.889684 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.889967 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.890010 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.921065 (+ 31055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.921067 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.921067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.921075 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.923311 (+ 2236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.923942 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.923943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.923943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.924005 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.924219 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.924220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.924220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.924372 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.924418 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.924653 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.924743 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:03:41.925014 31390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.886719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.0.0.1:40212)
0810 23:03:41.886860 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.886871 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.886931 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.887202 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.887204 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.887455 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.887553 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.887556 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.888537 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.888538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.889415 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.889417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.889950 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.889951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.890328 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.890664 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.890670 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.919961 (+ 29291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.922560 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.922561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.922562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.923749 (+ 1187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.924326 (+ 577us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.924327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.924327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.924398 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.924696 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.924697 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.924737 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.924790 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.924903 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260810 23:03:41.925230 31392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.887277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:47816)
0810 23:03:41.887418 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.887428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.887513 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.887690 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.887690 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.887853 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.887913 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.887914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.888805 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.888805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.889603 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.889604 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.890084 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.890085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.890143 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.919765 (+ 29622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.919772 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.920868 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.923403 (+ 2535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.923404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.923404 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.923857 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.924082 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.924082 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.924083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.924146 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.924468 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.924468 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.924528 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.924927 (+ 399us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.925019 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
I20260810 23:03:41.925357 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.886737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40212 (local address 127.23.56.125:45849)
0810 23:03:41.886794 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.886796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.887025 (+ 229us) server_negotiation.cc:407] Connection header received
0810 23:03:41.887053 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.887054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.887065 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.887152 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.887653 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.888418 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.889533 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.889827 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.889869 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.920184 (+ 30315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.920187 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.920187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.920197 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.922082 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.924197 (+ 2115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.924198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.924199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.924256 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.924468 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.924469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.924469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.924612 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.924670 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.925154 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.925241 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:03:41.926486 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.886993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.23.56.124:45181)
0810 23:03:41.887029 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.887031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.887142 (+ 111us) server_negotiation.cc:407] Connection header received
0810 23:03:41.887230 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.887231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.887239 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.887267 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.887745 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.888434 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.889454 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.889725 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.889760 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.920998 (+ 31238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.920999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.920999 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.921231 (+ 232us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.924092 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.925728 (+ 1636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.925729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.925729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.925781 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.925955 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.925955 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.925956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.926068 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.926107 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.926284 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.926412 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":20}
I20260810 23:03:41.926666 31391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.886933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.0.0.1:49288)
0810 23:03:41.887072 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.887081 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.887163 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.887322 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.887322 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.887602 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.887667 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.887669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.888513 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.888514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.889373 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.889376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.889831 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.889832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.889885 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.890331 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.890335 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.920268 (+ 29933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.925450 (+ 5182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.925451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.925452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.925651 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.925843 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.925843 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.925844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.925894 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.926145 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.926146 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.926178 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.926260 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.926584 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":75,"threads_started":1}
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403020, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.126@KRBTEST.COM for oryx/127.23.56.124@KRBTEST.COM
I20260810 23:03:41.934618 31367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.928020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.23.56.124:45181)
0810 23:03:41.928054 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.928056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.928077 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:03:41.928161 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.928162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.928172 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.928215 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.928507 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.929014 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.929681 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.929878 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.929952 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.931883 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.931885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.931885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.931892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.933405 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.933793 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.933794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.933794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.933851 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.934086 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.934086 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.934087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.934233 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.934289 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.934451 (+ 162us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.934527 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:03:41.934623 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.927979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.0.0.1:49298)
0810 23:03:41.928035 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.928046 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.928093 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.928228 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.928228 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.928377 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.928416 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.928417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.929089 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.929091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.929598 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.929600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.929949 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.929950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.930024 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.930417 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.930423 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.931787 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.933495 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.933496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.933497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.933725 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.933953 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.933954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.933955 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.934014 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.934295 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.934296 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.934339 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.934424 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.934529 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:03:41.935153 30553 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.56.124:45181: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74"
member_type: NON_VOTER
last_known_addr {
host: "127.23.56.124"
port: 45181
}
attrs {
promote: true
}
}
dest_uuid: "307f9e72b6a3437ebf590125864218af"
cas_config_opid_index: 5
I20260810 23:03:41.935412 30553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } }
I20260810 23:03:41.935775 31183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5e1ca015590a4be6ab5503c5e9b24f74 (127.23.56.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.936601 31183 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.56.124:45181
I20260810 23:03:41.936688 31183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.936785 31183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:41.937088 31287 master.cc:537] Master@127.23.56.124:45181 shutting down...
W20260810 23:03:41.937704 30537 consensus_peers.cc:487] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af -> Peer 5e1ca015590a4be6ab5503c5e9b24f74 (127.23.56.124:45181): Couldn't send request to peer 5e1ca015590a4be6ab5503c5e9b24f74. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 23:03:41.940552 31287 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:41.940634 31287 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:41.940673 31287 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: stopping tablet replica
I20260810 23:03:41.941927 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.937185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.0.0.1:40218)
0810 23:03:41.937821 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.937832 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.937871 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.938093 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.938094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.938096 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:41.938135 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.938137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.939175 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.939176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.940373 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.940376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.941636 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.941637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.941711 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.941737 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.941788 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.941833 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622}
I20260810 23:03:41.942171 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.937290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40218 (local address 127.23.56.125:45849)
0810 23:03:41.937792 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.937795 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.937921 (+ 126us) server_negotiation.cc:407] Connection header received
0810 23:03:41.937947 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.937948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.937975 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.938031 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:41.938207 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.939058 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.940550 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.941143 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.941221 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.941599 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.942064 (+ 465us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.942108 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474}
I20260810 23:03:41.942375 31238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 LEARNER]: Refusing update from remote peer 307f9e72b6a3437ebf590125864218af: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 23:03:41.942593 31184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Connected to new peer: Peer: permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 23:03:41.943176 31238 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 5e1ca015590a4be6ab5503c5e9b24f74 (127.23.56.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.944438 31271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307f9e72b6a3437ebf590125864218af. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.944526 31184 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: attempting to promote NON_VOTER 90454ac770814be18a301f882e97b0a0 to VOTER
I20260810 23:03:41.944536 31271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:03:41.944574 31271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: NON_VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: true } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.944612 31271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:03:41.944708 31184 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } }
I20260810 23:03:41.945327 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.945113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.0.0.1:49300)
0810 23:03:41.945243 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.945291 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260810 23:03:41.945363 31238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 LEARNER]: Refusing update from remote peer 307f9e72b6a3437ebf590125864218af: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 23:03:41.945755 31183 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Connected to new peer: Peer: permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 23:03:41.945816 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.945689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.0.0.1:49310)
0810 23:03:41.945748 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.945776 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:45181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:03:41.946020 30537 consensus_peers.cc:487] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af -> Peer 5e1ca015590a4be6ab5503c5e9b24f74 (127.23.56.124:45181): Couldn't send request to peer 5e1ca015590a4be6ab5503c5e9b24f74. Status: Network error: Client connection negotiation failed: client connection to 127.23.56.124:45181: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:03:41.946390 31182 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 90454ac770814be18a301f882e97b0a0 (127.23.56.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.947467 31183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.947553 31183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:41.947456 31182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.947613 31182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:41.947721 31238 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 90454ac770814be18a301f882e97b0a0 (127.23.56.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.948542 31397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:41.948628 31397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:41.955777 31287 master.cc:552] Master@127.23.56.124:45181 shutdown complete.
I20260810 23:03:41.955842 31287 master_runner.cc:305] Clearing existing system tablet
I20260810 23:03:41.955973 31287 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:41.956035 31287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:41.956653 31287 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:41.957183 31400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.957304 31287 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:41.957353 31287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:41.957382 31287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal
uuid: "5e1ca015590a4be6ab5503c5e9b24f74"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "d71fc43fb3d0f92a17be9f326ba04646"
server_key_iv: "0d85c946e7b4c144a6eab33b3f7b5534"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.957988 31287 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:41.959271 31287 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 23:03:41.959349 31287 log.cc:1196] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 23:03:41.959548 31287 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: Deleting consensus metadata
I20260810 23:03:41.959689 31287 master_runner.cc:315] Copying system tablet from 127.23.56.126:40607
I20260810 23:03:41.960397 31287 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: tablet copy: Beginning tablet copy session from remote peer at address 127.23.56.126:40607
I20260810 23:03:41.966543 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.960833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.23.56.126:40607)
0810 23:03:41.960881 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:41.960883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:41.961016 (+ 133us) server_negotiation.cc:407] Connection header received
0810 23:03:41.961096 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:41.961097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:41.961108 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:41.961158 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:41.961457 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.961953 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.962643 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.962841 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.962925 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.963934 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:41.963935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:41.963936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:41.963943 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:41.965360 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.965737 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.965739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.965739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.965800 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:41.966018 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:41.966019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:41.966020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:41.966132 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:41.966210 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:41.966362 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 23:03:41.966441 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:03:41.966543 31406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:41.960795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:47826)
0810 23:03:41.960929 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:41.960939 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:41.961032 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:41.961160 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:41.961161 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:41.961336 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:41.961381 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.961383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.962037 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.962039 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.962559 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:41.962561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:41.962928 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:41.962929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:41.962993 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:41.963493 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:41.963499 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:41.963820 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:41.965462 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.965464 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.965465 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.965679 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.965877 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:41.965878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:41.965878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:41.965941 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:41.966202 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:41.966203 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:41.966249 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 23:03:41.966336 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:03:41.966443 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
I20260810 23:03:41.967106 30577 tablet_copy_service.cc:140] P 307f9e72b6a3437ebf590125864218af: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5e1ca015590a4be6ab5503c5e9b24f74 ({username='oryx', principal='oryx/127.23.56.124@KRBTEST.COM'} at 127.0.0.1:47826)
I20260810 23:03:41.967216 30577 tablet_copy_service.cc:161] P 307f9e72b6a3437ebf590125864218af: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5e1ca015590a4be6ab5503c5e9b24f74 at {username='oryx', principal='oryx/127.23.56.124@KRBTEST.COM'} at 127.0.0.1:47826: session id = 5e1ca015590a4be6ab5503c5e9b24f74-00000000000000000000000000000000
I20260810 23:03:41.967682 30577 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 23:03:41.968154 31287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 23:03:41.969076 31287 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: tablet copy: Starting download of 0 data blocks...
I20260810 23:03:41.969235 31287 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: tablet copy: Starting download of 1 WAL segments...
I20260810 23:03:41.969923 31287 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 23:03:41.970705 30577 tablet_copy_service.cc:342] P 307f9e72b6a3437ebf590125864218af: Request end of tablet copy session 5e1ca015590a4be6ab5503c5e9b24f74-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.56.124@KRBTEST.COM'} at 127.0.0.1:47826
I20260810 23:03:41.970759 30577 tablet_copy_service.cc:433] P 307f9e72b6a3437ebf590125864218af: ending tablet copy session 5e1ca015590a4be6ab5503c5e9b24f74-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5e1ca015590a4be6ab5503c5e9b24f74
I20260810 23:03:41.973459 31287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:41.974612 31407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.974658 31410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:41.975000 31408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:41.975147 31287 server_base.cc:1034] running on GCE node
I20260810 23:03:41.975278 31287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:41.975407 31287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:41.976528 31287 hybrid_clock.cc:648] HybridClock initialized: now 1786403021976524 us; error 22 us; skew 500 ppm
Aug 10 23:03:41 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403021, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:41.977896 31287 init.cc:377] Logged in from keytab as oryx/127.23.56.124@KRBTEST.COM (short username oryx)
I20260810 23:03:41.978185 31287 webserver.cc:460] Webserver started at http://127.23.56.124:39441/ using document root <none> and password file <none>
I20260810 23:03:41.978307 31287 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:41.978348 31287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:41.978967 31287 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:41.979549 31419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.979696 31287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:41.979744 31287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal
uuid: "5e1ca015590a4be6ab5503c5e9b24f74"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "d71fc43fb3d0f92a17be9f326ba04646"
server_key_iv: "0d85c946e7b4c144a6eab33b3f7b5534"
server_key_version: "encryptionkey@0"
I20260810 23:03:41.979806 31287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:41.993350 31287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:41.995071 31287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:41.998242 31287 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.124:45181
I20260810 23:03:41.998522 31475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.124:45181 every 8 connection(s)
I20260810 23:03:41.998641 31287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
I20260810 23:03:41.999387 31477 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:03:41.999783 31477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: Bootstrap starting.
I20260810 23:03:42.002174 31477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:03:42.002418 31477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: Bootstrap complete.
I20260810 23:03:42.002629 31477 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } }
I20260810 23:03:42.002717 31477 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5e1ca015590a4be6ab5503c5e9b24f74, State: Initialized, Role: LEARNER
I20260810 23:03:42.002763 31477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } }
I20260810 23:03:42.002938 31477 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:42.002933 31480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:42.003250 31480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:03:42.004522 31495 catalog_manager.cc:1259] Loaded cluster ID: f51bfab748804bfb9d9da94266c4c310
I20260810 23:03:42.004727 31495 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: loading cluster ID for follower catalog manager: success
I20260810 23:03:42.005811 31495 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: acquiring CA information for follower catalog manager: success
I20260810 23:03:42.006084 31495 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:03:42.048645 31394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:42.048875 31394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:42.048913 31394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:42.050138 31394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:42.050202 31394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:42.050249 31394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:42.051931 31394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab
--principal=oryx/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:43395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=37367
--tserver_master_addrs=127.23.56.126:40607,127.23.56.125:45849,127.23.56.124:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:42.052194 31394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:42.052429 31394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:42.058633 31501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:42.058607 31502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:42.058622 31504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:42.058866 31394 server_base.cc:1034] running on GCE node
I20260810 23:03:42.059160 31394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:42.059412 31394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:42.060582 31394 hybrid_clock.cc:648] HybridClock initialized: now 1786403022060567 us; error 44 us; skew 500 ppm
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:42.063207 31394 init.cc:377] Logged in from keytab as oryx/127.23.56.65@KRBTEST.COM (short username oryx)
I20260810 23:03:42.063651 31394 webserver.cc:460] Webserver started at http://127.23.56.65:37367/ using document root <none> and password file <none>
I20260810 23:03:42.063905 31394 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:42.063995 31394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:42.065326 31394 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:42.065877 31512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:42.066071 31394 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:42.066166 31394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "12146b8872474a2ab2c251e990776763"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "1bd76ffced16f07bf7ff710e76003dda"
server_key_iv: "306a88ff55b8b8c4b946912e76c20e41"
server_key_version: "encryptionkey@0"
I20260810 23:03:42.066506 31394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:42.078612 31394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:42.080717 31394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:42.080868 31394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:42.081100 31394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:42.081409 31394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:42.081459 31394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:42.081502 31394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:42.081529 31394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:42.087844 31394 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:43395
I20260810 23:03:42.087895 31636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:43395 every 8 connection(s)
I20260810 23:03:42.088234 31394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:42.092197 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 31394
I20260810 23:03:42.092353 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 30774
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:42.099805 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.089039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:46255 (local address 127.23.56.126:40607)
0810 23:03:42.089563 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.089565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.089576 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:03:42.089932 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.089934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.089947 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.089988 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.090789 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.091379 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.092780 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.093188 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.093238 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.096099 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.096101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.096102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.096109 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.097770 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.098318 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.098320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.098320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.098384 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.098982 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.098984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.098984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.099123 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.099191 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.099609 (+ 418us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.099696 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500}
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.124@KRBTEST.COM
I20260810 23:03:42.100368 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.088935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.65:46255)
0810 23:03:42.089120 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.089162 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.089827 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.090066 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.090067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.090481 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:42.090692 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.090696 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.091486 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.091487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.092678 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.092680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.093466 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.093467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.093599 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.094206 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:42.094216 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:42.095996 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:42.097889 (+ 1893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.097896 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.097898 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.098219 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.098746 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.098746 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.098747 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.098892 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.099247 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:42.099254 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:42.099426 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.099934 (+ 508us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.100119 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260810 23:03:42.101061 31638 heartbeater.cc:347] Connected to a master server at 127.23.56.126:40607
I20260810 23:03:42.101152 31638 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:42.101325 31638 heartbeater.cc:511] Master 127.23.56.126:40607 requested a full tablet report, sending...
I20260810 23:03:42.101823 30553 ts_manager.cc:194] Re-registered known tserver with Master: 12146b8872474a2ab2c251e990776763 (127.23.56.65:43395)
I20260810 23:03:42.102155 30553 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:46255
I20260810 23:03:42.103816 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.089413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:43781 (local address 127.23.56.124:45181)
0810 23:03:42.089748 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.089750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.089757 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:03:42.089981 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.089982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.089997 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.090063 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.090807 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.091848 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.097126 (+ 5278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.098092 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.098168 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.100351 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.100353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.100354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.100363 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.102452 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.102843 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.102845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.102845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.102909 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.103210 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.103211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.103212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.103356 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.103412 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.103608 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.103678 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":58,"threads_started":1}
I20260810 23:03:42.103971 31647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.089320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.23.56.65:43781)
0810 23:03:42.089461 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.089472 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.089830 (+ 358us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.090067 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.090068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.090543 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:42.090693 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.090699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.091925 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.091926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.096739 (+ 4813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.096742 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.098228 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.098229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.098306 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.098738 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:42.098743 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:42.100238 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:42.102540 (+ 2302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.102541 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.102542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.102752 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.102987 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.102996 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.102997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.103083 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.103431 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:42.103432 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:42.103493 (+ 61us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.103586 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.103749 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260810 23:03:42.104446 31639 heartbeater.cc:347] Connected to a master server at 127.23.56.124:45181
I20260810 23:03:42.104501 31639 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:42.104555 31639 heartbeater.cc:511] Master 127.23.56.124:45181 requested a full tablet report, sending...
I20260810 23:03:42.105033 31440 ts_manager.cc:194] Registered new tserver with Master: 12146b8872474a2ab2c251e990776763 (127.23.56.65:43395)
I20260810 23:03:42.112286 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:42249
--local_ip_for_outbound_sockets=127.23.56.66
--tserver_master_addrs=127.23.56.126:40607
--webserver_port=35919
--webserver_interface=127.23.56.66
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab
--principal=oryx/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.56.126:40607,127.23.56.125:45849
--tserver_master_addrs=127.23.56.126:40607,127.23.56.125:45849,127.23.56.124:45181 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.125@KRBTEST.COM
I20260810 23:03:42.117009 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.089504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:59111 (local address 127.23.56.125:45849)
0810 23:03:42.089569 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.089571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.089582 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:03:42.089915 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.089916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.089928 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.089997 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.090806 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.091593 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.111842 (+ 20249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.112163 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.112246 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.114255 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.114258 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.114258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.114270 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.115787 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.116173 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.116174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.116175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.116234 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.116450 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.116450 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.116451 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.116584 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.116655 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.116824 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.116910 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 23:03:42.117038 31646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.088937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.23.56.65:59111)
0810 23:03:42.089188 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.089198 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.089809 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.090059 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.090064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.090481 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:42.090693 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.090699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.091687 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.091688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.111664 (+ 19976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.111670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.112285 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.112287 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.112371 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.112923 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:42.112928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:42.114152 (+ 1224us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:42.115891 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.115892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.115893 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.116079 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.116312 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.116313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.116314 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.116378 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.116658 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:42.116659 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:42.116706 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.116786 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.116912 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":24,"thread_start_us":107,"threads_started":1}
I20260810 23:03:42.117370 31640 heartbeater.cc:347] Connected to a master server at 127.23.56.125:45849
I20260810 23:03:42.117453 31640 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:42.117529 31640 heartbeater.cc:511] Master 127.23.56.125:45849 requested a full tablet report, sending...
I20260810 23:03:42.118000 31226 ts_manager.cc:194] Registered new tserver with Master: 12146b8872474a2ab2c251e990776763 (127.23.56.65:43395)
W20260810 23:03:42.197638 31649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:42.197887 31649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:42.197925 31649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:42.199108 31649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:42.199199 31649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:42.199245 31649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:42.200788 31649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab
--principal=oryx/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:42249
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=35919
--tserver_master_addrs=127.23.56.126:40607,127.23.56.125:45849,127.23.56.124:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:42.201069 31649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:42.201323 31649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:42.206934 31658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:42.206944 31655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:42.206944 31654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:42.207170 31649 server_base.cc:1034] running on GCE node
I20260810 23:03:42.207472 31649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:42.207726 31649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:42.208890 31649 hybrid_clock.cc:648] HybridClock initialized: now 1786403022208870 us; error 50 us; skew 500 ppm
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:42.211350 31649 init.cc:377] Logged in from keytab as oryx/127.23.56.66@KRBTEST.COM (short username oryx)
I20260810 23:03:42.211838 31649 webserver.cc:460] Webserver started at http://127.23.56.66:35919/ using document root <none> and password file <none>
I20260810 23:03:42.212074 31649 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:42.212157 31649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:42.213454 31649 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:42.214768 31668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:42.214960 31649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:42.215059 31649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "9e2e115077a142aba3475eccdcbd1890"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "ccadc71f8f83e6b435d445c6bcb008db"
server_key_iv: "48ed06a7e71e419afe561c8a64dbb0c7"
server_key_version: "encryptionkey@0"
I20260810 23:03:42.215462 31649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:42.233922 31649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:42.235777 31649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:42.235942 31649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:42.236177 31649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:42.236502 31649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:42.236567 31649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:42.236609 31649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:42.236639 31649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:42.242897 31649 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:42249
I20260810 23:03:42.242954 31795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:42249 every 8 connection(s)
I20260810 23:03:42.243307 31649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:42.248145 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 31649
I20260810 23:03:42.248323 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 30922
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.124@KRBTEST.COM
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.125@KRBTEST.COM
I20260810 23:03:42.256233 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.244491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:36773 (local address 127.23.56.125:45849)
0810 23:03:42.244548 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.244550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.244559 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:42.245416 (+ 857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.245417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.245431 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.245466 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.246290 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.247055 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.249134 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.249427 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.249504 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.252376 (+ 2872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.252378 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.252378 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.252386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.254459 (+ 2073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.255104 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.255105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.255105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.255189 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.255597 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.255598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.255598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.255730 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.255772 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.256056 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.256140 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 23:03:42.256302 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.244060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:58011 (local address 127.23.56.126:40607)
0810 23:03:42.244114 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.244117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.244382 (+ 265us) server_negotiation.cc:407] Connection header received
0810 23:03:42.245389 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.245391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.245403 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.245463 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.246274 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.246776 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.248288 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.248597 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.248642 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.251978 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.251979 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.251980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.251988 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.254766 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.255321 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.255322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.255322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.255376 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.255627 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.255628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.255628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.255777 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.255830 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.256065 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.256159 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 23:03:42.256589 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.244094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:34799 (local address 127.23.56.124:45181)
0810 23:03:42.244143 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.244145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.244396 (+ 251us) server_negotiation.cc:407] Connection header received
0810 23:03:42.245460 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.245461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.245478 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.245528 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.246285 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.247066 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.248343 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.248706 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.248771 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.252368 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.252369 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.252369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.252377 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.254246 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.255105 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.255106 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.255106 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.255188 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.255493 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.255494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.255494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.255636 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.255695 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.256396 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.256507 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:03:42.256750 31801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.244044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.66:58011)
0810 23:03:42.244236 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.244278 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.245278 (+ 1000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.245537 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.245542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.245967 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:42.246174 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.246179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.246888 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.246889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.248171 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.248174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.248730 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.248731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.248863 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.249461 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:42.249472 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:42.251863 (+ 2391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:42.254934 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.254935 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.254935 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.255255 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.255451 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.255451 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.255452 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.255546 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.255865 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:42.255866 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:42.255916 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.256390 (+ 474us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.256518 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":4,"thread_start_us":108,"threads_started":1}
I20260810 23:03:42.256757 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.244044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.23.56.66:34799)
0810 23:03:42.244324 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.244335 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.245307 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.245588 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.245589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.246017 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:42.246175 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.246182 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.247154 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.247155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.248171 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.248174 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.248879 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.248880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.248970 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.249962 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:42.249968 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:42.252272 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:42.254593 (+ 2321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.254593 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.254594 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.255030 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.255265 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.255266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.255266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.255418 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.255722 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:42.255729 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:42.255897 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.256022 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.256190 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":16,"thread_start_us":98,"threads_started":1}
I20260810 23:03:42.257095 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:35465
--local_ip_for_outbound_sockets=127.23.56.67
--tserver_master_addrs=127.23.56.126:40607
--webserver_port=41725
--webserver_interface=127.23.56.67
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab
--principal=oryx/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.56.126:40607,127.23.56.125:45849
--tserver_master_addrs=127.23.56.126:40607,127.23.56.125:45849,127.23.56.124:45181 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:03:42.257232 31797 heartbeater.cc:347] Connected to a master server at 127.23.56.124:45181
I20260810 23:03:42.257320 31797 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:42.257519 31797 heartbeater.cc:511] Master 127.23.56.124:45181 requested a full tablet report, sending...
I20260810 23:03:42.257535 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.244374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.23.56.66:36773)
0810 23:03:42.244495 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.244505 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.245307 (+ 802us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.245588 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.245589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.245977 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:42.246175 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.246181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.248268 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.248269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.249065 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.249067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.249560 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.249561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.249625 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.249963 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:42.249968 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:42.252272 (+ 2304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:42.254554 (+ 2282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.254563 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.254565 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.255008 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.255266 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.255267 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.255267 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.255517 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.255816 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:42.255816 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:42.255897 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.257323 (+ 1426us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.257431 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260810 23:03:42.258066 31440 ts_manager.cc:194] Registered new tserver with Master: 9e2e115077a142aba3475eccdcbd1890 (127.23.56.66:42249)
I20260810 23:03:42.258168 31798 heartbeater.cc:347] Connected to a master server at 127.23.56.125:45849
I20260810 23:03:42.258216 31798 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:42.258286 31798 heartbeater.cc:511] Master 127.23.56.125:45849 requested a full tablet report, sending...
I20260810 23:03:42.258617 31226 ts_manager.cc:194] Registered new tserver with Master: 9e2e115077a142aba3475eccdcbd1890 (127.23.56.66:42249)
I20260810 23:03:42.259080 31796 heartbeater.cc:347] Connected to a master server at 127.23.56.126:40607
I20260810 23:03:42.259150 31796 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:42.259212 31796 heartbeater.cc:511] Master 127.23.56.126:40607 requested a full tablet report, sending...
I20260810 23:03:42.259415 30553 ts_manager.cc:194] Re-registered known tserver with Master: 9e2e115077a142aba3475eccdcbd1890 (127.23.56.66:42249)
I20260810 23:03:42.259764 30553 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:58011
W20260810 23:03:42.340969 31805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:42.341199 31805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:42.341245 31805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:42.342407 31805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:42.342475 31805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:42.342521 31805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:42.344213 31805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab
--principal=oryx/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:35465
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=41725
--tserver_master_addrs=127.23.56.126:40607,127.23.56.125:45849,127.23.56.124:45181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:42.344501 31805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:42.344738 31805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:42.349937 31813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:42.349936 31811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:42.350003 31810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:42.350109 31805 server_base.cc:1034] running on GCE node
I20260810 23:03:42.350387 31805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:42.350628 31805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:42.351787 31805 hybrid_clock.cc:648] HybridClock initialized: now 1786403022351777 us; error 43 us; skew 500 ppm
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:42.354825 31805 init.cc:377] Logged in from keytab as oryx/127.23.56.67@KRBTEST.COM (short username oryx)
I20260810 23:03:42.355343 31805 webserver.cc:460] Webserver started at http://127.23.56.67:41725/ using document root <none> and password file <none>
I20260810 23:03:42.355605 31805 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:42.355690 31805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:42.357067 31805 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:42.357710 31821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:42.357874 31805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:42.357986 31805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "7530190c7d1445818b024acf973c940f"
format_stamp: "Formatted at 2026-08-10 23:03:41 on dist-test-slave-1xrh"
server_key: "61ccaedd6b80cd61a82c25dc71159f6e"
server_key_iv: "4c0b96467f46a1c51b7a27284f78b79c"
server_key_version: "encryptionkey@0"
I20260810 23:03:42.358347 31805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:42.372632 31805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:42.374987 31805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:42.375185 31805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:42.375478 31805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:42.375797 31805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:42.375854 31805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:42.375897 31805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:42.375926 31805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:42.382055 31805 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:35465
I20260810 23:03:42.382128 31954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:35465 every 8 connection(s)
I20260810 23:03:42.382472 31805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:42.382741 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 31805
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.125@KRBTEST.COM
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.124@KRBTEST.COM
I20260810 23:03:42.392866 31967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.383929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47832 (local address 127.23.56.126:40607)
0810 23:03:42.384086 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.384088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.384097 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:42.384127 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.384128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.384141 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.384217 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.385143 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.385659 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.386740 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.387045 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.387103 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.388712 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.388713 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.388714 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.388720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.391180 (+ 2460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.391578 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.391579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.391580 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.391639 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.391843 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.391845 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.391845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.392043 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.392151 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.392658 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.392750 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
Aug 10 23:03:42 dist-test-slave-1xrh krb5kdc[30504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403022, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:42.396982 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.383763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:60689 (local address 127.23.56.124:45181)
0810 23:03:42.383839 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.383842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.386161 (+ 2319us) server_negotiation.cc:407] Connection header received
0810 23:03:42.386193 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.386195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.386206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.386262 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.386849 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.387617 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.389144 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.389440 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.389480 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.393860 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.393861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.393862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.393868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.395696 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.396184 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.396185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.396185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.396241 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.396437 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.396438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.396438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.396567 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.396608 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.396814 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.396891 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:03:42.396996 31283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.383732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:43515 (local address 127.23.56.125:45849)
0810 23:03:42.383800 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.383803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.386279 (+ 2476us) server_negotiation.cc:407] Connection header received
0810 23:03:42.386366 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.386367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.386378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.386407 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.386930 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.387616 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.389153 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.389448 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.389506 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.393793 (+ 4287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.393794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.393795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.393802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.395426 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.395918 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.395920 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.395920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.395971 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.396266 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.396267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.396268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.396399 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.396443 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.396814 (+ 371us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.396891 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:03:42.397511 31965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.383656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.23.56.67:43515)
0810 23:03:42.386228 (+ 2572us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.386236 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.386300 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.386461 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.386462 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.386786 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:42.386846 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.386848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.387758 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.387759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.389042 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.389045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.389594 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.389595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.389715 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.390513 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:42.390517 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:42.393681 (+ 3164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:42.395526 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.395534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.395536 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.395834 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.396059 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.396060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.396060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.396187 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.396489 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:42.396496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:42.396656 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.397090 (+ 434us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.397293 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2551,"mutex_wait_us":200,"thread_start_us":82,"threads_started":1}
I20260810 23:03:42.397920 31962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.383239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.23.56.67:60689)
0810 23:03:42.386065 (+ 2826us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.386074 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.386148 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.386315 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.386316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.386479 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:42.386732 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.386738 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.387754 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.387754 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.389041 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.389042 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.389571 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.389572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.389715 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.390451 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:42.390455 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:42.393767 (+ 3312us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:42.395887 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.395888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.395889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.396104 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.396313 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.396314 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.396314 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.396365 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.396655 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:42.396656 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:42.396686 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.397433 (+ 747us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.397534 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2800,"mutex_wait_us":55,"thread_start_us":92,"threads_started":1}
I20260810 23:03:42.398172 31957 heartbeater.cc:347] Connected to a master server at 127.23.56.125:45849
I20260810 23:03:42.398267 31957 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:42.398455 31957 heartbeater.cc:511] Master 127.23.56.125:45849 requested a full tablet report, sending...
I20260810 23:03:42.398612 31956 heartbeater.cc:347] Connected to a master server at 127.23.56.124:45181
I20260810 23:03:42.398662 31956 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:42.398736 31956 heartbeater.cc:511] Master 127.23.56.124:45181 requested a full tablet report, sending...
I20260810 23:03:42.398897 31226 ts_manager.cc:194] Registered new tserver with Master: 7530190c7d1445818b024acf973c940f (127.23.56.67:35465)
I20260810 23:03:42.399010 30598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.383695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:54221 (local address 127.23.56.126:40607)
0810 23:03:42.383831 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.383833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.384519 (+ 686us) server_negotiation.cc:407] Connection header received
0810 23:03:42.385839 (+ 1320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.385840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.385850 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.385938 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.386957 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.388020 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.389144 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.389440 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.389491 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.394224 (+ 4733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.394225 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.394226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.394232 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.396478 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.397356 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.397357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.397357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.397415 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.397651 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.397652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.397652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.397775 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.397818 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.398844 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.398920 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260810 23:03:42.399025 31440 ts_manager.cc:194] Registered new tserver with Master: 7530190c7d1445818b024acf973c940f (127.23.56.67:35465)
I20260810 23:03:42.399384 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.383239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.67:54221)
0810 23:03:42.384376 (+ 1137us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.384424 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.385730 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.386028 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.386037 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.386411 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:42.386824 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.386826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.388113 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.388114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.389039 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.389042 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.389573 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.389573 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.389715 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.390413 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:42.390433 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:42.394110 (+ 3677us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:42.396573 (+ 2463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.396574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.396575 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.397111 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.397495 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:42.397495 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:42.397496 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:42.397575 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:42.397936 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:42.397937 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:42.398637 (+ 700us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.399168 (+ 531us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.399269 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1093,"mutex_wait_us":6,"thread_start_us":96,"threads_started":1}
I20260810 23:03:42.397734 30553 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47832:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:42.400079 30553 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:42.400763 31955 heartbeater.cc:347] Connected to a master server at 127.23.56.126:40607
I20260810 23:03:42.400820 31955 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:42.400909 31955 heartbeater.cc:511] Master 127.23.56.126:40607 requested a full tablet report, sending...
I20260810 23:03:42.401223 30552 ts_manager.cc:194] Re-registered known tserver with Master: 7530190c7d1445818b024acf973c940f (127.23.56.67:35465)
I20260810 23:03:42.401687 30552 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:54221
I20260810 23:03:42.403853 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.400160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.0.0.1:49324)
0810 23:03:42.400195 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.400207 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.400257 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.400410 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.400411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.400415 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:42.400473 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.400474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.401325 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.401326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.402428 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.402430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.403552 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.403553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.403635 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.403638 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.403686 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.403735 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260810 23:03:42.404137 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.400257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49324 (local address 127.23.56.124:45181)
0810 23:03:42.400291 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.400293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.400300 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:03:42.400319 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.400320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.400340 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.400401 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:42.400553 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.401170 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.402688 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.403354 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.403421 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.403504 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.404016 (+ 512us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.404058 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:03:42.404383 31450 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 23:03:42.404561 31480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 307f9e72b6a3437ebf590125864218af. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:42.404665 31480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: This master's current role is: LEARNER
I20260810 23:03:42.405145 31184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: NON_VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: true } } }
I20260810 23:03:42.405253 31184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:42.406306 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.384227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49320 (local address 127.23.56.124:45181)
0810 23:03:42.392262 (+ 8035us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.392264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.392271 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:03:42.392290 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.392290 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.392300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.393098 (+ 798us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.393137 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.393814 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.394843 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.395178 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.395484 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.396960 (+ 1476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.396961 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.396962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.396969 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.404159 (+ 7190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.404905 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.404906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.404907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.404970 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.405193 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.405194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.405194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.405319 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.405769 (+ 450us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.405812 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.406218 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8017,"thread_start_us":54,"threads_started":1}
I20260810 23:03:42.407157 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.384161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40234 (local address 127.23.56.125:45849)
0810 23:03:42.388006 (+ 3845us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.388009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.388018 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:42.388040 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.388041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.388053 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.388126 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.390488 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.391266 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.392212 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.392510 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.392574 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.401875 (+ 9301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.401876 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.401877 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.401884 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.405575 (+ 3691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.405938 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.405939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.405939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.406002 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.406203 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.406204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.406204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.406326 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.406521 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.406947 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.407035 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3824,"thread_start_us":59,"threads_started":1}
I20260810 23:03:42.408560 31182 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: attempting to promote NON_VOTER 5e1ca015590a4be6ab5503c5e9b24f74 to VOTER
I20260810 23:03:42.408713 31182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } }
I20260810 23:03:42.409210 31238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Refusing update from remote peer 307f9e72b6a3437ebf590125864218af: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260810 23:03:42.409500 31449 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 LEARNER]: Refusing update from remote peer 307f9e72b6a3437ebf590125864218af: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260810 23:03:42.409797 31184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 23:03:42.409911 31184 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [LEADER]: Connected to new peer: Peer: permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20260810 23:03:42.410645 31183 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 5e1ca015590a4be6ab5503c5e9b24f74 (127.23.56.124) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:42.411294 31238 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 5e1ca015590a4be6ab5503c5e9b24f74 (127.23.56.124) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:42.412405 31397 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:42.412514 31397 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:42.412920 31181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.407361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:35465 (local address 127.0.0.1:41016)
0810 23:03:42.407629 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.407638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.407676 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.408477 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.408478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.408479 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:42.408566 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.408567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.409939 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.409941 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.411186 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.411189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.412692 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.412693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.412782 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.412785 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.412814 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.412848 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251}
I20260810 23:03:42.413146 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.407925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41016 (local address 127.23.56.67:35465)
0810 23:03:42.408332 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.408336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.408344 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:03:42.408371 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.408373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.408385 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.408480 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:42.408649 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.409839 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.411664 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.412530 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.412887 (+ 357us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.412958 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.412987 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.413051 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":66,"threads_started":1}
I20260810 23:03:42.413372 31983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.407795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:42249 (local address 127.0.0.1:45124)
0810 23:03:42.408173 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.408183 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.408229 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.409014 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.409015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.409017 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:42.409054 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.409055 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.410570 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.410571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.411767 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.411770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.412692 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.412693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.412760 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.412766 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.412798 (+ 32us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.412842 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260810 23:03:42.413343 31183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:42.413516 31184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:42.413568 31183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:42.413589 31184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:42.413853 31880 tablet_service.cc:1467] Processing CreateTablet for tablet 806e1278a4dc4a32b80a86990535afce (DEFAULT_TABLE table=test-table [id=91bae7b9dfe34172a11b85a4e67ea27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:42.414129 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.407977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45124 (local address 127.23.56.66:42249)
0810 23:03:42.408665 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.408672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.408682 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:42.408732 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.408733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.408750 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.408949 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:42.409127 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.410461 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.411858 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.412602 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.413837 (+ 1235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.413925 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.413952 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.413996 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":92,"threads_started":1}
I20260810 23:03:42.414306 31880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 806e1278a4dc4a32b80a86990535afce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:42.414888 31720 tablet_service.cc:1467] Processing CreateTablet for tablet 806e1278a4dc4a32b80a86990535afce (DEFAULT_TABLE table=test-table [id=91bae7b9dfe34172a11b85a4e67ea27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:42.415223 31720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 806e1278a4dc4a32b80a86990535afce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:42.417824 31182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:42.417915 31182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:42.418224 31450 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, 5e1ca015590a4be6ab5503c5e9b24f74 (127.23.56.124) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:42.418458 31988 tablet_bootstrap.cc:492] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: Bootstrap starting.
I20260810 23:03:42.418756 31398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.407450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:43395 (local address 127.0.0.1:36244)
0810 23:03:42.408045 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.408055 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.408092 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.415526 (+ 7434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.415528 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.415530 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:42.415568 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.415569 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.416524 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.416525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.417391 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.417394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.418559 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.418561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.418618 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.418623 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.418654 (+ 31us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.418697 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579}
I20260810 23:03:42.419209 31988 tablet_bootstrap.cc:654] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:42.419252 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.407666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36244 (local address 127.23.56.65:43395)
0810 23:03:42.415319 (+ 7653us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.415324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.415336 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:03:42.415375 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.415377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.415394 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.415637 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:42.415707 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.416407 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.417501 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.418441 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.418950 (+ 509us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.419045 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.419074 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.419115 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7609,"thread_start_us":139,"threads_started":1}
I20260810 23:03:42.419576 31989 tablet_bootstrap.cc:492] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: Bootstrap starting.
I20260810 23:03:42.419549 31480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "307f9e72b6a3437ebf590125864218af" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:42.419646 31480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:42.419632 31988 log.cc:824] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:42.420291 31989 tablet_bootstrap.cc:654] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:42.420264 31565 tablet_service.cc:1467] Processing CreateTablet for tablet 806e1278a4dc4a32b80a86990535afce (DEFAULT_TABLE table=test-table [id=91bae7b9dfe34172a11b85a4e67ea27c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:42.420666 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 806e1278a4dc4a32b80a86990535afce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:42.420737 31988 tablet_bootstrap.cc:492] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: No bootstrap required, opened a new log
I20260810 23:03:42.420815 31988 ts_tablet_manager.cc:1397] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 23:03:42.420907 31989 log.cc:824] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:42.422698 31989 tablet_bootstrap.cc:492] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: No bootstrap required, opened a new log
I20260810 23:03:42.422775 31989 ts_tablet_manager.cc:1397] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:03:42.423792 31988 raft_consensus.cc:348] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:42.423961 31988 raft_consensus.cc:374] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:42.424008 31988 raft_consensus.cc:729] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7530190c7d1445818b024acf973c940f, State: Initialized, Role: FOLLOWER
I20260810 23:03:42.424160 31988 consensus_queue.cc:260] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:42.424423 31988 ts_tablet_manager.cc:1428] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20260810 23:03:42.424964 31994 tablet_bootstrap.cc:492] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: Bootstrap starting.
I20260810 23:03:42.425357 31989 raft_consensus.cc:348] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:42.425525 31989 raft_consensus.cc:374] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:42.425570 31989 raft_consensus.cc:729] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2e115077a142aba3475eccdcbd1890, State: Initialized, Role: FOLLOWER
I20260810 23:03:42.425643 31994 tablet_bootstrap.cc:654] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:42.425693 31989 consensus_queue.cc:260] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:42.425972 31989 ts_tablet_manager.cc:1428] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 23:03:42.426010 31994 log.cc:824] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:42.426015 31796 heartbeater.cc:503] Master 127.23.56.126:40607 was elected leader, sending a full tablet report...
I20260810 23:03:42.426080 31955 heartbeater.cc:503] Master 127.23.56.126:40607 was elected leader, sending a full tablet report...
I20260810 23:03:42.426725 31994 tablet_bootstrap.cc:492] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: No bootstrap required, opened a new log
I20260810 23:03:42.426802 31994 ts_tablet_manager.cc:1397] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:03:42.428844 31994 raft_consensus.cc:348] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:42.428962 31994 raft_consensus.cc:374] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:42.428984 31994 raft_consensus.cc:729] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12146b8872474a2ab2c251e990776763, State: Initialized, Role: FOLLOWER
I20260810 23:03:42.429127 31994 consensus_queue.cc:260] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:42.429390 31994 ts_tablet_manager.cc:1428] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:42.429457 31638 heartbeater.cc:503] Master 127.23.56.126:40607 was elected leader, sending a full tablet report...
I20260810 23:03:42.441828 31996 raft_consensus.cc:482] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:42.441967 31996 raft_consensus.cc:504] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:42.442346 31996 leader_election.cc:288] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7530190c7d1445818b024acf973c940f (127.23.56.67:35465), 12146b8872474a2ab2c251e990776763 (127.23.56.65:43395)
I20260810 23:03:42.445437 31984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.442627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:44915 (local address 127.23.56.67:35465)
0810 23:03:42.442690 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.442693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.442702 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:42.442732 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.442733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.442743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.442802 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:42.443019 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.443571 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.444717 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.445082 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.445177 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.445186 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.445335 (+ 149us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.445372 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:03:42.445439 31802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.442511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:35465 (local address 127.23.56.66:44915)
0810 23:03:42.442581 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.442595 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.442642 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.442832 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.442834 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.442837 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:42.442914 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.442920 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.443666 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.443667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.444608 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.444610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.445177 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.445178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.445253 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.445257 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.445330 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.445368 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260810 23:03:42.445575 31800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.442599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:43395 (local address 127.23.56.66:37843)
0810 23:03:42.442719 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:42.442728 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:42.442756 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:42.443018 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:42.443019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:42.443021 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:42.443086 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.443087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.443766 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.443767 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.444792 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:42.444795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.445308 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.445309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:42.445386 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.445389 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:42.445459 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:03:42.445498 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 23:03:42.445575 31982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.442857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:37843 (local address 127.23.56.65:43395)
0810 23:03:42.442910 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.442911 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.442918 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:03:42.442937 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.442938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.442945 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.443014 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:42.443219 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.443682 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.444872 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.445226 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.445316 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.445324 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.445478 (+ 154us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.445515 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 23:03:42.446024 31586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806e1278a4dc4a32b80a86990535afce" candidate_uuid: "9e2e115077a142aba3475eccdcbd1890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12146b8872474a2ab2c251e990776763" is_pre_election: true
I20260810 23:03:42.446026 31902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806e1278a4dc4a32b80a86990535afce" candidate_uuid: "9e2e115077a142aba3475eccdcbd1890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7530190c7d1445818b024acf973c940f" is_pre_election: true
I20260810 23:03:42.446177 31586 raft_consensus.cc:2454] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2e115077a142aba3475eccdcbd1890 in term 0.
I20260810 23:03:42.446177 31902 raft_consensus.cc:2454] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e2e115077a142aba3475eccdcbd1890 in term 0.
I20260810 23:03:42.446415 31669 leader_election.cc:302] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12146b8872474a2ab2c251e990776763, 9e2e115077a142aba3475eccdcbd1890; no voters:
I20260810 23:03:42.446575 31996 raft_consensus.cc:2781] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:42.446632 31996 raft_consensus.cc:482] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:42.446660 31996 raft_consensus.cc:3037] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:42.447587 31996 raft_consensus.cc:504] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:42.447731 31996 leader_election.cc:288] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [CANDIDATE]: Term 1 election: Requested vote from peers 7530190c7d1445818b024acf973c940f (127.23.56.67:35465), 12146b8872474a2ab2c251e990776763 (127.23.56.65:43395)
I20260810 23:03:42.447875 31902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806e1278a4dc4a32b80a86990535afce" candidate_uuid: "9e2e115077a142aba3475eccdcbd1890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7530190c7d1445818b024acf973c940f"
I20260810 23:03:42.447896 31586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "806e1278a4dc4a32b80a86990535afce" candidate_uuid: "9e2e115077a142aba3475eccdcbd1890" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12146b8872474a2ab2c251e990776763"
I20260810 23:03:42.447956 31902 raft_consensus.cc:3037] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:42.447988 31586 raft_consensus.cc:3037] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:42.448680 31902 raft_consensus.cc:2454] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2e115077a142aba3475eccdcbd1890 in term 1.
I20260810 23:03:42.448804 31586 raft_consensus.cc:2454] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e2e115077a142aba3475eccdcbd1890 in term 1.
I20260810 23:03:42.448860 31670 leader_election.cc:302] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7530190c7d1445818b024acf973c940f, 9e2e115077a142aba3475eccdcbd1890; no voters:
I20260810 23:03:42.448975 31996 raft_consensus.cc:2781] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:42.449118 31996 raft_consensus.cc:686] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 1 LEADER]: Becoming Leader. State: Replica: 9e2e115077a142aba3475eccdcbd1890, State: Running, Role: LEADER
I20260810 23:03:42.449231 31996 consensus_queue.cc:237] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:42.450228 30552 catalog_manager.cc:5495] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9e2e115077a142aba3475eccdcbd1890 (127.23.56.66). New cstate: current_term: 1 leader_uuid: "9e2e115077a142aba3475eccdcbd1890" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } health_report { overall_health: UNKNOWN } } }
I20260810 23:03:42.458285 31985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.456183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45138 (local address 127.23.56.66:42249)
0810 23:03:42.456232 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.456234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.456242 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:03:42.456315 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.456316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.456332 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.456388 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:42.456468 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.456920 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.457627 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.457812 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.457862 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.457993 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:42.458087 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:42.458143 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.458176 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.458192 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 23:03:42.460150 31902 raft_consensus.cc:1261] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [term 1 FOLLOWER]: Refusing update from remote peer 9e2e115077a142aba3475eccdcbd1890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:42.460191 31586 raft_consensus.cc:1261] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [term 1 FOLLOWER]: Refusing update from remote peer 9e2e115077a142aba3475eccdcbd1890: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:42.460558 31999 consensus_queue.cc:1035] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:42.460692 31996 consensus_queue.cc:1035] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:42.464370 32006 mvcc.cc:205] Tried to move back new op lower bound from 7317106779993993216 to 7317106779952926720. Current Snapshot: MvccSnapshot[applied={T|T < 7317106779993993216}]
I20260810 23:03:42.464764 32003 mvcc.cc:205] Tried to move back new op lower bound from 7317106779993993216 to 7317106779952926720. Current Snapshot: MvccSnapshot[applied={T|T < 7317106779993993216 or (T in {7317106779993993216})}]
I20260810 23:03:42.465490 32004 mvcc.cc:205] Tried to move back new op lower bound from 7317106779993993216 to 7317106779952926720. Current Snapshot: MvccSnapshot[applied={T|T < 7317106779993993216}]
I20260810 23:03:42.466048 30553 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:42.466150 30553 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47832:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:42.467568 30553 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 307f9e72b6a3437ebf590125864218af: Sending DeleteTablet for 3 replicas of tablet 806e1278a4dc4a32b80a86990535afce
I20260810 23:03:42.468021 31565 tablet_service.cc:1514] Processing DeleteTablet for tablet 806e1278a4dc4a32b80a86990535afce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:42 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:36244
I20260810 23:03:42.468357 31880 tablet_service.cc:1514] Processing DeleteTablet for tablet 806e1278a4dc4a32b80a86990535afce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:42 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:41016
I20260810 23:03:42.468464 32013 tablet_replica.cc:330] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: stopping tablet replica
I20260810 23:03:42.468559 32013 raft_consensus.cc:2229] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:42.468658 32013 raft_consensus.cc:2258] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:42.469144 32013 ts_tablet_manager.cc:1905] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:42.469251 31720 tablet_service.cc:1514] Processing DeleteTablet for tablet 806e1278a4dc4a32b80a86990535afce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:42 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:45124
I20260810 23:03:42.469333 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 30519
I20260810 23:03:42.474018 32013 ts_tablet_manager.cc:1918] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:42.474114 32013 log.cc:1196] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/806e1278a4dc4a32b80a86990535afce
I20260810 23:03:42.474381 32013 ts_tablet_manager.cc:1939] T 806e1278a4dc4a32b80a86990535afce P 12146b8872474a2ab2c251e990776763: Deleting consensus metadata
W20260810 23:03:42.479163 31514 connection.cc:535] server connection from 127.0.0.1:36244 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:03:42.480448 32015 tablet_replica.cc:330] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: stopping tablet replica
I20260810 23:03:42.480600 32015 raft_consensus.cc:2229] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:42.480726 32015 raft_consensus.cc:2258] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:42.481218 32015 ts_tablet_manager.cc:1905] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:42.483001 32016 tablet_replica.cc:330] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: stopping tablet replica
I20260810 23:03:42.483114 32016 raft_consensus.cc:2229] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:42.483174 32015 ts_tablet_manager.cc:1918] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:42.483235 32016 raft_consensus.cc:2258] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:42.483254 32015 log.cc:1196] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/806e1278a4dc4a32b80a86990535afce
I20260810 23:03:42.483587 32015 ts_tablet_manager.cc:1939] T 806e1278a4dc4a32b80a86990535afce P 7530190c7d1445818b024acf973c940f: Deleting consensus metadata
W20260810 23:03:42.484025 31826 connection.cc:441] server connection from 127.0.0.1:41016 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:41016 (request call id 1) could send its response
I20260810 23:03:42.484416 32016 ts_tablet_manager.cc:1905] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:42.486233 32016 ts_tablet_manager.cc:1918] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:42.486382 32016 log.cc:1196] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/806e1278a4dc4a32b80a86990535afce
I20260810 23:03:42.486706 32016 ts_tablet_manager.cc:1939] T 806e1278a4dc4a32b80a86990535afce P 9e2e115077a142aba3475eccdcbd1890: Deleting consensus metadata
W20260810 23:03:42.487337 31669 connection.cc:441] server connection from 127.0.0.1:45124 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:45124 (request call id 1) could send its response
I20260810 23:03:42.491909 31648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.468575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49332 (local address 127.23.56.124:45181)
0810 23:03:42.468621 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.468623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.483257 (+ 14634us) server_negotiation.cc:407] Connection header received
0810 23:03:42.483361 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.483363 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.483375 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.483407 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.484192 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.484920 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.485987 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.486320 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.486360 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.488362 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.488364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.488364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.488373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.490411 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.490801 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.490803 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.490803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.490860 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.491064 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.491065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.491066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.491222 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.491314 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.491722 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.491799 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:03:42.492548 31973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.481686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49334 (local address 127.23.56.124:45181)
0810 23:03:42.481727 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.481729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.483451 (+ 1722us) server_negotiation.cc:407] Connection header received
0810 23:03:42.483656 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.483657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.483671 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.483701 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.485227 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.485925 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.487023 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.487335 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.487382 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.488804 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.488805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.488805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.488813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.491034 (+ 2221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.491408 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.491409 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.491409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.491461 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.491664 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.491665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.491665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.491786 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.491850 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.492341 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.492429 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:03:42.493357 31971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:42.481479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.23.56.125:45849)
0810 23:03:42.481522 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:42.481525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:42.481997 (+ 472us) server_negotiation.cc:407] Connection header received
0810 23:03:42.482038 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:42.482039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:42.482057 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:42.482123 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:42.482900 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.483702 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.488418 (+ 4716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:42.488768 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:42.488809 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:42.490411 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:42.490412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:42.490413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:42.490421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:42.492122 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.492508 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.492509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.492510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.492569 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:42.492801 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:42.492802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:42.492803 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:42.492938 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:42.492995 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:42.493169 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 23:03:42.493259 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:03:43.465058 32027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.464731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.65:35675)
0810 23:03:43.464910 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.465010 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":70,"threads_started":1}
W20260810 23:03:43.465324 31638 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:40607 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
I20260810 23:03:43.466223 32029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.465939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.66:40647)
0810 23:03:43.466096 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.466177 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
W20260810 23:03:43.466434 31796 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:40607 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
I20260810 23:03:43.467168 32030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.466724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.67:41437)
0810 23:03:43.467026 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.467100 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":73,"threads_started":1}
W20260810 23:03:43.467360 31955 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:40607 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
I20260810 23:03:43.967716 32031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 307f9e72b6a3437ebf590125864218af)
I20260810 23:03:43.967859 32031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } }
I20260810 23:03:43.968227 32031 leader_election.cc:288] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 307f9e72b6a3437ebf590125864218af (127.23.56.126:40607), 5e1ca015590a4be6ab5503c5e9b24f74 (127.23.56.124:45181)
I20260810 23:03:43.968665 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.968381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:48306)
0810 23:03:43.968542 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.968614 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":52,"threads_started":1}
I20260810 23:03:43.968683 32035 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 307f9e72b6a3437ebf590125864218af)
I20260810 23:03:43.968742 32035 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } }
W20260810 23:03:43.968869 31211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
I20260810 23:03:43.969138 32035 leader_election.cc:288] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 307f9e72b6a3437ebf590125864218af (127.23.56.126:40607), 90454ac770814be18a301f882e97b0a0 (127.23.56.125:45849)
I20260810 23:03:43.969300 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.969011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:48314)
0810 23:03:43.969202 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.969258 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":47,"threads_started":1}
W20260810 23:03:43.969403 31423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
I20260810 23:03:43.969579 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.969485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:48326)
0810 23:03:43.969528 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.969552 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 23:03:43.969656 31211 leader_election.cc:334] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 307f9e72b6a3437ebf590125864218af (127.23.56.126:40607): Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
I20260810 23:03:43.970008 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.969908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:48336)
0810 23:03:43.969959 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.969984 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 23:03:43.970079 31423 leader_election.cc:334] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 307f9e72b6a3437ebf590125864218af (127.23.56.126:40607): Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
I20260810 23:03:43.971910 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.968411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:45181 (local address 127.0.0.1:49340)
0810 23:03:43.968546 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.968558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:43.968623 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:43.968877 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:43.968878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:43.968881 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:43.968944 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:43.968947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:43.969949 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:43.969950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:43.971060 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:43.971063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:43.971669 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:43.971670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:43.971738 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:43.971742 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:43.971794 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 23:03:43.971863 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":31,"thread_start_us":51,"threads_started":1}
I20260810 23:03:43.971921 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.968598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49340 (local address 127.23.56.124:45181)
0810 23:03:43.968699 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:43.968703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:43.968731 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:03:43.968763 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:43.968764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:43.968777 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:43.968847 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:43.969038 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:43.969854 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:43.971193 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:43.971578 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:43.971641 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:43.971651 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:43.971814 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:03:43.971860 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":52,"threads_started":1}
I20260810 23:03:43.972236 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90454ac770814be18a301f882e97b0a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" is_pre_election: true
I20260810 23:03:43.972359 31450 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90454ac770814be18a301f882e97b0a0 in term 1.
I20260810 23:03:43.972513 31210 leader_election.cc:302] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e1ca015590a4be6ab5503c5e9b24f74, 90454ac770814be18a301f882e97b0a0; no voters: 307f9e72b6a3437ebf590125864218af
I20260810 23:03:43.972657 32031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 23:03:43.972718 32031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 307f9e72b6a3437ebf590125864218af)
I20260810 23:03:43.972728 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.969068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:45849 (local address 127.0.0.1:40258)
0810 23:03:43.969249 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.969259 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:43.969308 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:43.969845 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:43.969847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:43.969849 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:43.969900 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:43.969903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:43.970659 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:43.970660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:43.971604 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:43.971606 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:43.972471 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:43.972472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:43.972527 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:43.972529 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:03:43.972594 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:03:43.972638 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":40,"threads_started":1}
I20260810 23:03:43.972761 32031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:03:43.972769 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.969145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.23.56.125:45849)
0810 23:03:43.969655 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:43.969657 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:43.969666 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:43.969688 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:43.969689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:43.969699 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:43.969771 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:43.970004 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:43.970580 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:43.971716 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:43.972353 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:43.972429 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:43.972439 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:43.972598 (+ 159us) server_negotiation.cc:299] Negotiation successful
0810 23:03:43.972639 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":486,"thread_start_us":89,"threads_started":1}
I20260810 23:03:43.972977 31238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "90454ac770814be18a301f882e97b0a0" is_pre_election: true
I20260810 23:03:43.973594 32031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } }
I20260810 23:03:43.973711 31238 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5e1ca015590a4be6ab5503c5e9b24f74 in current term 2: Already voted for candidate 90454ac770814be18a301f882e97b0a0 in this term.
I20260810 23:03:43.973738 32031 leader_election.cc:288] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [CANDIDATE]: Term 2 election: Requested vote from peers 307f9e72b6a3437ebf590125864218af (127.23.56.126:40607), 5e1ca015590a4be6ab5503c5e9b24f74 (127.23.56.124:45181)
I20260810 23:03:43.973883 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.973804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:48352)
0810 23:03:43.973835 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.973863 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260810 23:03:43.973922 31422 leader_election.cc:302] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5e1ca015590a4be6ab5503c5e9b24f74; no voters: 307f9e72b6a3437ebf590125864218af, 90454ac770814be18a301f882e97b0a0
I20260810 23:03:43.973922 31450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "90454ac770814be18a301f882e97b0a0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5e1ca015590a4be6ab5503c5e9b24f74"
I20260810 23:03:43.974032 32035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:03:43.974205 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:43.974135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:48360)
0810 23:03:43.974165 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:43.974189 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260810 23:03:43.974309 31211 leader_election.cc:334] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 307f9e72b6a3437ebf590125864218af (127.23.56.126:40607): Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
I20260810 23:03:43.974907 32035 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:03:43.975554 31450 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90454ac770814be18a301f882e97b0a0 in term 2.
I20260810 23:03:43.976014 31210 leader_election.cc:302] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5e1ca015590a4be6ab5503c5e9b24f74, 90454ac770814be18a301f882e97b0a0; no voters: 307f9e72b6a3437ebf590125864218af
I20260810 23:03:43.976109 32031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:03:43.976272 32031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [term 2 LEADER]: Becoming Leader. State: Replica: 90454ac770814be18a301f882e97b0a0, State: Running, Role: LEADER
I20260810 23:03:43.976361 32031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } }
I20260810 23:03:43.976714 32042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90454ac770814be18a301f882e97b0a0. Latest consensus state: current_term: 2 leader_uuid: "90454ac770814be18a301f882e97b0a0" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:43.976814 32042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:43.976967 32044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:43.977456 32044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:43.977545 32044 catalog_manager.cc:1259] Loaded cluster ID: f51bfab748804bfb9d9da94266c4c310
I20260810 23:03:43.977581 32044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:43.977744 32044 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:43.977826 32044 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Loaded TSK: 0
I20260810 23:03:43.977967 32044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:03:44.031678 31225 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:44.033341 31225 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:44.034391 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.034261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:48376)
0810 23:03:44.034321 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.034356 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 23:03:44.034404 31450 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [term 2 FOLLOWER]: Refusing update from remote peer 90454ac770814be18a301f882e97b0a0: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 23:03:44.034741 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.034658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.0.0.1:48386)
0810 23:03:44.034698 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.034721 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 23:03:44.034694 32042 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260810 23:03:44.034835 31211 consensus_peers.cc:487] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 -> Peer 307f9e72b6a3437ebf590125864218af (127.23.56.126:40607): Couldn't send request to peer 307f9e72b6a3437ebf590125864218af. Status: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 23:03:44.035379 32035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90454ac770814be18a301f882e97b0a0. Latest consensus state: current_term: 2 leader_uuid: "90454ac770814be18a301f882e97b0a0" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:44.035462 32035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:44.035851 32045 mvcc.cc:205] Tried to move back new op lower bound from 7317106786442780672 to 7317106786207739904. Current Snapshot: MvccSnapshot[applied={T|T < 7317106786442780672}]
I20260810 23:03:44.035967 32031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "90454ac770814be18a301f882e97b0a0" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:44.036074 32031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:44.036108 32035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "90454ac770814be18a301f882e97b0a0" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "307f9e72b6a3437ebf590125864218af" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 40607 } } peers { permanent_uuid: "90454ac770814be18a301f882e97b0a0" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 45849 } attrs { promote: false } } peers { permanent_uuid: "5e1ca015590a4be6ab5503c5e9b24f74" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 45181 } attrs { promote: false } } }
I20260810 23:03:44.036197 32035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e1ca015590a4be6ab5503c5e9b24f74 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:03:44.036340 32046 mvcc.cc:205] Tried to move back new op lower bound from 7317106786442780672 to 7317106786207739904. Current Snapshot: MvccSnapshot[applied={T|T < 7317106786442780672}]
I20260810 23:03:44.040973 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.038064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45148 (local address 127.23.56.66:42249)
0810 23:03:44.038209 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:44.038211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:44.038220 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:44.038246 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:44.038247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:44.038257 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:44.038352 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:44.038461 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.039053 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.040084 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.040606 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.040685 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.040699 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:44.040862 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:03:44.040907 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260810 23:03:44.040963 32034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.037822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:42249 (local address 127.0.0.1:45148)
0810 23:03:44.037880 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.037891 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:44.037944 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:44.038356 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:44.038356 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:44.038358 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:44.038385 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.038386 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.039153 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.039154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.039986 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.039994 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.040695 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.040696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.040770 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.040775 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:03:44.040853 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:03:44.040898 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":35}
I20260810 23:03:44.041411 31720 tablet_service.cc:1467] Processing CreateTablet for tablet 3069978c68a54b19866de0162666d8bf (DEFAULT_TABLE table=test-table [id=4614404120f64e32a48aaee593a8b8f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:44.041491 32050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.037873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:43395 (local address 127.0.0.1:36256)
0810 23:03:44.038008 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.038019 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:44.038080 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:44.038582 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:44.038583 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:44.038585 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:44.038635 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.038637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.039352 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.039353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.040462 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.040464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.041263 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.041264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.041322 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.041326 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:44.041372 (+ 46us) client_negotiation.cc:240] Negotiation successful
0810 23:03:44.041411 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260810 23:03:44.041586 31720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3069978c68a54b19866de0162666d8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:44.041705 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.038070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36256 (local address 127.23.56.65:43395)
0810 23:03:44.038429 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:44.038431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:44.038439 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:03:44.038466 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:44.038467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:44.038477 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:44.038537 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:44.038702 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.039283 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.040558 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.041159 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.041219 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.041233 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:44.041589 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 23:03:44.041636 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":70,"threads_started":1}
I20260810 23:03:44.042086 32033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.037822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:35465 (local address 127.0.0.1:41032)
0810 23:03:44.037919 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.037927 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:44.037958 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:44.038332 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:44.038333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:44.038335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:44.038383 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.038385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.040039 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.040040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.041068 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.041070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.041832 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.041833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.041891 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.041895 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:44.041986 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:03:44.042024 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":18}
I20260810 23:03:44.042105 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.037885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.23.56.67:35465)
0810 23:03:44.038201 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:44.038204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:44.038213 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:44.038242 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:44.038242 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:44.038255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:44.038317 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:44.039192 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.039940 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.041170 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.041732 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.041828 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.041840 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:44.041987 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 23:03:44.042026 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":68,"threads_started":1}
I20260810 23:03:44.043073 32056 tablet_bootstrap.cc:492] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: Bootstrap starting.
I20260810 23:03:44.043318 31565 tablet_service.cc:1467] Processing CreateTablet for tablet 3069978c68a54b19866de0162666d8bf (DEFAULT_TABLE table=test-table [id=4614404120f64e32a48aaee593a8b8f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:44.043385 31880 tablet_service.cc:1467] Processing CreateTablet for tablet 3069978c68a54b19866de0162666d8bf (DEFAULT_TABLE table=test-table [id=4614404120f64e32a48aaee593a8b8f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:44.043435 31565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3069978c68a54b19866de0162666d8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:44.043511 31880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3069978c68a54b19866de0162666d8bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:44.043551 32056 tablet_bootstrap.cc:654] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:44.044227 32056 tablet_bootstrap.cc:492] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: No bootstrap required, opened a new log
I20260810 23:03:44.044274 32056 ts_tablet_manager.cc:1397] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:44.044401 32056 raft_consensus.cc:348] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:44.044462 32056 raft_consensus.cc:374] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:44.044503 32056 raft_consensus.cc:729] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e2e115077a142aba3475eccdcbd1890, State: Initialized, Role: FOLLOWER
I20260810 23:03:44.044569 32056 consensus_queue.cc:260] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:44.044581 32058 tablet_bootstrap.cc:492] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763: Bootstrap starting.
I20260810 23:03:44.044697 32056 ts_tablet_manager.cc:1428] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:44.045126 32058 tablet_bootstrap.cc:654] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:44.045588 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.045016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.66:43921)
0810 23:03:44.045500 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.045550 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":83,"threads_started":1}
I20260810 23:03:44.045606 32060 tablet_bootstrap.cc:492] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: Bootstrap starting.
I20260810 23:03:44.045626 31798 heartbeater.cc:503] Master 127.23.56.125:45849 was elected leader, sending a full tablet report...
I20260810 23:03:44.045987 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.045872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.66:57269)
0810 23:03:44.045927 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.045959 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:03:44.046187 32060 tablet_bootstrap.cc:654] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:44.046782 32060 tablet_bootstrap.cc:492] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: No bootstrap required, opened a new log
I20260810 23:03:44.046824 32060 ts_tablet_manager.cc:1397] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:44.046833 32058 tablet_bootstrap.cc:492] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763: No bootstrap required, opened a new log
I20260810 23:03:44.046882 32058 ts_tablet_manager.cc:1397] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:03:44.046974 32060 raft_consensus.cc:348] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:44.047032 32060 raft_consensus.cc:374] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:44.047045 32060 raft_consensus.cc:729] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7530190c7d1445818b024acf973c940f, State: Initialized, Role: FOLLOWER
I20260810 23:03:44.047050 32058 raft_consensus.cc:348] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:44.047096 32060 consensus_queue.cc:260] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:44.047151 32058 raft_consensus.cc:374] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:44.047186 32058 raft_consensus.cc:729] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12146b8872474a2ab2c251e990776763, State: Initialized, Role: FOLLOWER
I20260810 23:03:44.047276 32060 ts_tablet_manager.cc:1428] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:44.047262 32058 consensus_queue.cc:260] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:44.047428 32058 ts_tablet_manager.cc:1428] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:44.047663 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.047417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.67:53531)
0810 23:03:44.047586 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.047623 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":99,"threads_started":1}
I20260810 23:03:44.047930 32069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.047726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.65:41827)
0810 23:03:44.047857 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.047892 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260810 23:03:44.048599 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.048497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.67:37611)
0810 23:03:44.048550 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.048574 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 23:03:44.051645 32065 raft_consensus.cc:482] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:44.051729 32065 raft_consensus.cc:504] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:44.052027 32065 leader_election.cc:288] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e2e115077a142aba3475eccdcbd1890 (127.23.56.66:42249), 12146b8872474a2ab2c251e990776763 (127.23.56.65:43395)
I20260810 23:03:44.055207 32070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.052196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:43395 (local address 127.23.56.67:59785)
0810 23:03:44.052361 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.052371 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:44.052424 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:44.052557 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:44.052558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:44.052560 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:44.052600 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.052601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.053381 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.053382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.054288 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.054291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.054931 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.054932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.055004 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.055008 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:44.055056 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 23:03:44.055091 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":84,"thread_start_us":98,"threads_started":1}
I20260810 23:03:44.055219 32055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.052370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:59785 (local address 127.23.56.65:43395)
0810 23:03:44.052408 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:44.052410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:44.052420 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:44.052477 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:44.052477 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:44.052487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:44.052552 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:44.052663 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.053305 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.054369 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.054851 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.054907 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.054914 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:44.055071 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 23:03:44.055109 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:03:44.055302 32053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.052262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:54337 (local address 127.23.56.66:42249)
0810 23:03:44.052310 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:44.052311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:44.052322 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:03:44.052375 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:44.052376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:44.052388 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:44.052443 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:44.052567 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.053259 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.054462 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.054911 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.054978 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.054988 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:44.055184 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 23:03:44.055229 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:03:44.055317 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.052196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:42249 (local address 127.23.56.67:54337)
0810 23:03:44.052251 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.052262 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:44.052312 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:44.052447 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:44.052448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:44.052450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:44.052503 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.052505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.053332 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.053333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.054385 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:44.054387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.055008 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.055009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:44.055087 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.055090 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:44.055174 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:03:44.055215 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":21}
I20260810 23:03:44.055817 31586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3069978c68a54b19866de0162666d8bf" candidate_uuid: "7530190c7d1445818b024acf973c940f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12146b8872474a2ab2c251e990776763" is_pre_election: true
I20260810 23:03:44.055945 31586 raft_consensus.cc:2454] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7530190c7d1445818b024acf973c940f in term 0.
I20260810 23:03:44.055902 31746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3069978c68a54b19866de0162666d8bf" candidate_uuid: "7530190c7d1445818b024acf973c940f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2e115077a142aba3475eccdcbd1890" is_pre_election: true
I20260810 23:03:44.056021 31746 raft_consensus.cc:2454] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7530190c7d1445818b024acf973c940f in term 0.
I20260810 23:03:44.056120 31822 leader_election.cc:302] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12146b8872474a2ab2c251e990776763, 7530190c7d1445818b024acf973c940f; no voters:
I20260810 23:03:44.056257 32065 raft_consensus.cc:2781] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:44.056303 32065 raft_consensus.cc:482] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:44.056334 32065 raft_consensus.cc:3037] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:44.056964 32065 raft_consensus.cc:504] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:44.057107 32065 leader_election.cc:288] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [CANDIDATE]: Term 1 election: Requested vote from peers 9e2e115077a142aba3475eccdcbd1890 (127.23.56.66:42249), 12146b8872474a2ab2c251e990776763 (127.23.56.65:43395)
I20260810 23:03:44.057260 31586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3069978c68a54b19866de0162666d8bf" candidate_uuid: "7530190c7d1445818b024acf973c940f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12146b8872474a2ab2c251e990776763"
I20260810 23:03:44.057250 31746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3069978c68a54b19866de0162666d8bf" candidate_uuid: "7530190c7d1445818b024acf973c940f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e2e115077a142aba3475eccdcbd1890"
I20260810 23:03:44.057327 31586 raft_consensus.cc:3037] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:44.057348 31746 raft_consensus.cc:3037] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:44.057917 31586 raft_consensus.cc:2454] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7530190c7d1445818b024acf973c940f in term 1.
I20260810 23:03:44.058023 31746 raft_consensus.cc:2454] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7530190c7d1445818b024acf973c940f in term 1.
I20260810 23:03:44.058087 31822 leader_election.cc:302] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12146b8872474a2ab2c251e990776763, 7530190c7d1445818b024acf973c940f; no voters:
I20260810 23:03:44.058166 32065 raft_consensus.cc:2781] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:44.058297 32065 raft_consensus.cc:686] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 1 LEADER]: Becoming Leader. State: Replica: 7530190c7d1445818b024acf973c940f, State: Running, Role: LEADER
I20260810 23:03:44.058360 31957 heartbeater.cc:503] Master 127.23.56.125:45849 was elected leader, sending a full tablet report...
I20260810 23:03:44.058501 32065 consensus_queue.cc:237] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } }
I20260810 23:03:44.058606 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.058499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.67:36357)
0810 23:03:44.058551 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.058581 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 23:03:44.059219 31226 catalog_manager.cc:5495] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f reported cstate change: term changed from 0 to 1, leader changed from <none> to 7530190c7d1445818b024acf973c940f (127.23.56.67). New cstate: current_term: 1 leader_uuid: "7530190c7d1445818b024acf973c940f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7530190c7d1445818b024acf973c940f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 } health_report { overall_health: UNKNOWN } } }
I20260810 23:03:44.068405 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.066253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41034 (local address 127.23.56.67:35465)
0810 23:03:44.066310 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:44.066311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:44.066320 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:44.066386 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:44.066387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:44.066404 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:44.066465 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:44.066579 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.067013 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.067769 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:44.067973 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:44.068028 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:44.068161 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:44.068249 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:44.068300 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:44.068337 (+ 37us) server_negotiation.cc:299] Negotiation successful
0810 23:03:44.068355 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 23:03:44.069412 31586 raft_consensus.cc:1261] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [term 1 FOLLOWER]: Refusing update from remote peer 7530190c7d1445818b024acf973c940f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:44.069432 31746 raft_consensus.cc:1261] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [term 1 FOLLOWER]: Refusing update from remote peer 7530190c7d1445818b024acf973c940f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:44.069648 32065 consensus_queue.cc:1035] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [LEADER]: Connected to new peer: Peer: permanent_uuid: "12146b8872474a2ab2c251e990776763" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:44.069787 32072 consensus_queue.cc:1035] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e2e115077a142aba3475eccdcbd1890" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 42249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:44.070410 31640 heartbeater.cc:503] Master 127.23.56.125:45849 was elected leader, sending a full tablet report...
I20260810 23:03:44.070730 32069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.070378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.65:42633)
0810 23:03:44.070659 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.070698 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263}
I20260810 23:03:44.070744 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.070633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.66:33507)
0810 23:03:44.070685 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.070712 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:03:44.071789 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.071644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.67:44623)
0810 23:03:44.071732 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.071760 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 23:03:44.072080 32069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.071941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.65:33523)
0810 23:03:44.072022 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.072050 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:03:44.072515 32062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.072423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.66:40397)
0810 23:03:44.072458 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.072486 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260810 23:03:44.072525 32067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:44.072062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:40607 (local address 127.23.56.67:33927)
0810 23:03:44.072470 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.072503 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:40607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
I20260810 23:03:44.074687 31225 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:44.074775 31225 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40250:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:44.076097 31225 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 90454ac770814be18a301f882e97b0a0: Sending DeleteTablet for 3 replicas of tablet 3069978c68a54b19866de0162666d8bf
I20260810 23:03:44.076408 31565 tablet_service.cc:1514] Processing DeleteTablet for tablet 3069978c68a54b19866de0162666d8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:44 UTC) from {username='oryx', principal='oryx/127.23.56.125@KRBTEST.COM'} at 127.0.0.1:36256
I20260810 23:03:44.076408 31880 tablet_service.cc:1514] Processing DeleteTablet for tablet 3069978c68a54b19866de0162666d8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:44 UTC) from {username='oryx', principal='oryx/127.23.56.125@KRBTEST.COM'} at 127.0.0.1:41032
I20260810 23:03:44.076536 31720 tablet_service.cc:1514] Processing DeleteTablet for tablet 3069978c68a54b19866de0162666d8bf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:44 UTC) from {username='oryx', principal='oryx/127.23.56.125@KRBTEST.COM'} at 127.0.0.1:45148
I20260810 23:03:44.076642 32088 tablet_replica.cc:330] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: stopping tablet replica
I20260810 23:03:44.076701 32088 raft_consensus.cc:2229] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:44.076790 32088 raft_consensus.cc:2258] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:44.076916 32087 tablet_replica.cc:330] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763: stopping tablet replica
I20260810 23:03:44.076973 32087 raft_consensus.cc:2229] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:44.077023 32087 raft_consensus.cc:2258] T 3069978c68a54b19866de0162666d8bf P 12146b8872474a2ab2c251e990776763 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:44.077118 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 31394
I20260810 23:03:44.077306 32088 ts_tablet_manager.cc:1905] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:44.079330 32088 ts_tablet_manager.cc:1918] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:44.079403 32088 log.cc:1196] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/3069978c68a54b19866de0162666d8bf
I20260810 23:03:44.079674 32088 ts_tablet_manager.cc:1939] T 3069978c68a54b19866de0162666d8bf P 7530190c7d1445818b024acf973c940f: Deleting consensus metadata
I20260810 23:03:44.080014 32089 tablet_replica.cc:330] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: stopping tablet replica
I20260810 23:03:44.080025 31209 catalog_manager.cc:4841] TS 7530190c7d1445818b024acf973c940f (127.23.56.67:35465): tablet 3069978c68a54b19866de0162666d8bf (table test-table [id=4614404120f64e32a48aaee593a8b8f7]) successfully deleted
I20260810 23:03:44.080080 32089 raft_consensus.cc:2229] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:44.080127 32089 raft_consensus.cc:2258] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:44.080466 32089 ts_tablet_manager.cc:1905] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:44.081792 32089 ts_tablet_manager.cc:1918] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:44.081863 32089 log.cc:1196] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/3069978c68a54b19866de0162666d8bf
I20260810 23:03:44.082145 32089 ts_tablet_manager.cc:1939] T 3069978c68a54b19866de0162666d8bf P 9e2e115077a142aba3475eccdcbd1890: Deleting consensus metadata
I20260810 23:03:44.083283 31210 catalog_manager.cc:4841] TS 9e2e115077a142aba3475eccdcbd1890 (127.23.56.66:42249): tablet 3069978c68a54b19866de0162666d8bf (table test-table [id=4614404120f64e32a48aaee593a8b8f7]) successfully deleted
W20260810 23:03:44.086851 32033 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:44.086694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:43395 (local address 127.0.0.1:36260)
0810 23:03:44.086755 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:44.086788 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:43395: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:03:44.087066 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 31649
W20260810 23:03:44.087071 31208 catalog_manager.cc:4584] TS 12146b8872474a2ab2c251e990776763 (127.23.56.65:43395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3069978c68a54b19866de0162666d8bf: Network error: Client connection negotiation failed: client connection to 127.23.56.65:43395: connect: Connection reset by peer (error 104)
I20260810 23:03:44.098812 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 31805
I20260810 23:03:44.107267 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 31073
I20260810 23:03:44.114367 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 31287
2026-08-10T23:03:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4627 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:44 dist-test-slave-1xrh krb5kdc[32092](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:44 dist-test-slave-1xrh krb5kdc[32092](info): set up 2 sockets
Aug 10 23:03:44 dist-test-slave-1xrh krb5kdc[32092](info): commencing operation
krb5kdc: starting...
W20260810 23:03:45.233178 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:45 dist-test-slave-1xrh krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.126.keytab.
Entry for principal oryx/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.126.keytab.
I20260810 23:03:45.362521 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:46733
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.126.keytab
--principal=oryx/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:46733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:45.444026 32106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:45.444223 32106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:45.444247 32106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:45.445327 32106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:45.445361 32106 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 23:03:45.445374 32106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:45.445387 32106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:45.445400 32106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:45.445412 32106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:45.446849 32106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:46733
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.126.keytab
--principal=oryx/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:46733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:45.447114 32106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:45.447330 32106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:45.452195 32116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:45.452315 32113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:45.452273 32106 server_base.cc:1034] running on GCE node
W20260810 23:03:45.452455 32112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:45.452689 32106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:45.452924 32106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:45.454068 32106 hybrid_clock.cc:648] HybridClock initialized: now 1786403025454052 us; error 30 us; skew 500 ppm
Aug 10 23:03:45 dist-test-slave-1xrh krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:45.460965 32106 init.cc:377] Logged in from keytab as oryx/127.23.56.126@KRBTEST.COM (short username oryx)
I20260810 23:03:45.461454 32106 webserver.cc:460] Webserver started at http://127.23.56.126:37829/ using document root <none> and password file <none>
I20260810 23:03:45.461686 32106 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:45.461755 32106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:45.461913 32106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:45.462973 32106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "cceb5ac5bba040c79df96c5dde00f84c"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "4a6ec90f05f4bd6b608f139d0234816d"
server_key_iv: "71fc5798c37df1c02da54eb4147bd37d"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.463379 32106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "cceb5ac5bba040c79df96c5dde00f84c"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "4a6ec90f05f4bd6b608f139d0234816d"
server_key_iv: "71fc5798c37df1c02da54eb4147bd37d"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.464777 32106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:45.465536 32123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:45.465740 32106 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:45.465842 32106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "cceb5ac5bba040c79df96c5dde00f84c"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "4a6ec90f05f4bd6b608f139d0234816d"
server_key_iv: "71fc5798c37df1c02da54eb4147bd37d"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.465926 32106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:45.473397 32106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:45.475116 32106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:45.475337 32106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:45.480216 32106 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:46733
I20260810 23:03:45.480268 32198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:46733 every 8 connection(s)
I20260810 23:03:45.480594 32106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:45.481184 32199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:45.483503 32199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c: Bootstrap starting.
I20260810 23:03:45.484107 32199 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:45.484406 32199 log.cc:824] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:45.485078 32199 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c: No bootstrap required, opened a new log
I20260810 23:03:45.487080 32199 raft_consensus.cc:348] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cceb5ac5bba040c79df96c5dde00f84c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46733 } }
I20260810 23:03:45.487296 32199 raft_consensus.cc:374] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:45.487350 32199 raft_consensus.cc:729] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cceb5ac5bba040c79df96c5dde00f84c, State: Initialized, Role: FOLLOWER
I20260810 23:03:45.487535 32199 consensus_queue.cc:260] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cceb5ac5bba040c79df96c5dde00f84c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46733 } }
I20260810 23:03:45.487649 32199 raft_consensus.cc:388] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:45.487694 32199 raft_consensus.cc:482] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:45.487740 32199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:45.487726 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 32106
I20260810 23:03:45.487864 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:45.488096 23777 external_mini_cluster.cc:1428] Setting key 6044e3252fde97414aa539b7281eab47
I20260810 23:03:45.488448 32199 raft_consensus.cc:504] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cceb5ac5bba040c79df96c5dde00f84c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46733 } }
I20260810 23:03:45.488569 32199 leader_election.cc:302] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cceb5ac5bba040c79df96c5dde00f84c; no voters:
I20260810 23:03:45.488735 32199 leader_election.cc:288] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:45.488765 32204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:45.488880 32204 raft_consensus.cc:686] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [term 1 LEADER]: Becoming Leader. State: Replica: cceb5ac5bba040c79df96c5dde00f84c, State: Running, Role: LEADER
I20260810 23:03:45.488968 32204 consensus_queue.cc:237] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cceb5ac5bba040c79df96c5dde00f84c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46733 } }
I20260810 23:03:45.489002 32199 sys_catalog.cc:564] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:45.489346 32207 sys_catalog.cc:455] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [sys.catalog]: SysCatalogTable state changed. Reason: New leader cceb5ac5bba040c79df96c5dde00f84c. Latest consensus state: current_term: 1 leader_uuid: "cceb5ac5bba040c79df96c5dde00f84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cceb5ac5bba040c79df96c5dde00f84c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46733 } } }
I20260810 23:03:45.489351 32206 sys_catalog.cc:455] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cceb5ac5bba040c79df96c5dde00f84c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cceb5ac5bba040c79df96c5dde00f84c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46733 } } }
I20260810 23:03:45.489486 32206 sys_catalog.cc:458] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:45.489439 32207 sys_catalog.cc:458] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:45.489895 32212 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:45.490425 32212 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:45.492089 32212 catalog_manager.cc:1347] Generated new cluster ID: 09c1cb1f91a44a55ab7bf19e58d5e560
I20260810 23:03:45.492146 32212 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:03:45 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:45.500963 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.488667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52420 (local address 127.23.56.126:46733)
0810 23:03:45.490374 (+ 1707us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:45.490384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:45.490417 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:03:45.490533 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:45.490541 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:45.490596 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:45.490736 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:45.491503 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.492858 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.493900 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.494515 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.494611 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.497366 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:45.497371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:45.497371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:45.497384 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:45.499333 (+ 1949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.499709 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.499712 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.499713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.499833 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.500035 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.500036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.500036 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.500341 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:45.500526 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:45.500593 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 23:03:45.500729 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1644,"thread_start_us":77,"threads_started":1}
I20260810 23:03:45.503541 32212 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:45.504094 32212 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:45.508796 32212 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c: Generated new TSK 0
I20260810 23:03:45.508980 32212 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab.
Entry for principal oryx/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab.
I20260810 23:03:45.535787 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:46733
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab
--principal=oryx/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:45.617175 32229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:45.617375 32229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:45.617399 32229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:45.618534 32229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:45.618573 32229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:45.618594 32229 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:03:45.618613 32229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:45.620188 32229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab
--principal=oryx/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.56.126:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:45.620446 32229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:45.620676 32229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:45.625630 32236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:45.625640 32240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:45.625794 32229 server_base.cc:1034] running on GCE node
W20260810 23:03:45.625640 32235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:45.626036 32229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:45.626271 32229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:45.627429 32229 hybrid_clock.cc:648] HybridClock initialized: now 1786403025627420 us; error 46 us; skew 500 ppm
Aug 10 23:03:45 dist-test-slave-1xrh krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:45.629882 32229 init.cc:377] Logged in from keytab as oryx/127.23.56.65@KRBTEST.COM (short username oryx)
I20260810 23:03:45.630352 32229 webserver.cc:460] Webserver started at http://127.23.56.65:33793/ using document root <none> and password file <none>
I20260810 23:03:45.630599 32229 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:45.630668 32229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:45.630825 32229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:45.631916 32229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "571a317f77bd45c5bd9b66fb7a478503"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "d2a1bea93b90c758e4f2bc49fc51b6a1"
server_key_iv: "2e897ce14d71081f88148e06c50782b0"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.632275 32229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "571a317f77bd45c5bd9b66fb7a478503"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "d2a1bea93b90c758e4f2bc49fc51b6a1"
server_key_iv: "2e897ce14d71081f88148e06c50782b0"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.633582 32229 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:45.634320 32247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:45.634625 32229 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:45.634727 32229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "571a317f77bd45c5bd9b66fb7a478503"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "d2a1bea93b90c758e4f2bc49fc51b6a1"
server_key_iv: "2e897ce14d71081f88148e06c50782b0"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.634800 32229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:45.657271 32229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:45.659070 32229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:45.659252 32229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:45.659840 32229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:45.659890 32229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:45.659924 32229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:45.659942 32229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:03:45 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:45.667492 32229 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:33051
I20260810 23:03:45.667910 32229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:45.668407 32389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:33051 every 8 connection(s)
I20260810 23:03:45.670068 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.660244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:35375 (local address 127.23.56.126:46733)
0810 23:03:45.660313 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:45.660315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:45.660605 (+ 290us) server_negotiation.cc:407] Connection header received
0810 23:03:45.661304 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:45.661305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:45.661315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:45.661390 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:45.662306 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.662873 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.663990 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.664207 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.664292 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.666772 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:45.666774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:45.666775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:45.666784 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:45.668678 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.669140 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.669141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.669141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.669191 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.669477 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.669478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.669478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.669585 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:45.669659 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:45.669932 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 23:03:45.670003 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:03:45.670328 32258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.660250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46733 (local address 127.23.56.65:35375)
0810 23:03:45.660416 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:45.660482 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:45.661200 (+ 718us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:45.661495 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:45.661500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:45.661900 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:45.662200 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.662207 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.662966 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.662967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.663882 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.663885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.664306 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.664307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.664431 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.664914 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:45.664922 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:45.666688 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:45.668752 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.668758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.668759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.669066 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.669256 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.669257 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.669257 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.669389 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.669659 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:45.669665 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:45.669797 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 23:03:45.669906 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 23:03:45.670035 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
I20260810 23:03:45.670826 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 32229
I20260810 23:03:45.670964 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:45.670981 32390 heartbeater.cc:347] Connected to a master server at 127.23.56.126:46733
I20260810 23:03:45.671056 32390 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:45.671249 32390 heartbeater.cc:511] Master 127.23.56.126:46733 requested a full tablet report, sending...
I20260810 23:03:45.671245 23777 external_mini_cluster.cc:1428] Setting key f88b948311baed72ced89663d67b9c8b
I20260810 23:03:45.671813 32142 ts_manager.cc:194] Registered new tserver with Master: 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051)
I20260810 23:03:45.672686 32142 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:35375
I20260810 23:03:45.679790 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.672175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:38353 (local address 127.23.56.126:46733)
0810 23:03:45.673786 (+ 1611us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:45.673788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:45.673799 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:03:45.673820 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:45.673821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:45.673832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:45.673896 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:45.674217 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.674939 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.676071 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.676243 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.676295 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.677318 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:45.677319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:45.677319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:45.677325 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:45.678649 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.679049 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.679050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.679050 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.679096 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.679343 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.679343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.679344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.679438 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:45.679496 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:45.679669 (+ 173us) server_negotiation.cc:299] Negotiation successful
0810 23:03:45.679729 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1583}
I20260810 23:03:45.679893 32397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.672173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46733 (local address 127.23.56.65:38353)
0810 23:03:45.672312 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:45.672324 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:45.672378 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:45.673919 (+ 1541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:45.673920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:45.674078 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:45.674139 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.674140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.675057 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.675058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.675971 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.675974 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.676320 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.676321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.676397 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.676809 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:45.676815 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:45.677232 (+ 417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:45.678734 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.678735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.678735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.678963 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.679205 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.679206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.679207 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.679276 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.679504 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:45.679505 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:45.679554 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:03:45.679655 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:03:45.679760 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for oryx/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab.
Entry for principal oryx/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab.
I20260810 23:03:45.699007 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:46733
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab
--principal=oryx/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:45.781210 32401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:45.781417 32401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:45.781440 32401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:45.782611 32401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:45.782653 32401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:45.782675 32401 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:03:45.782732 32401 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:45.784276 32401 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab
--principal=oryx/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.56.126:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32401
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:45.784511 32401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:45.784759 32401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:45.789835 32409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:45.789942 32412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:45.789836 32410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:45.790024 32401 server_base.cc:1034] running on GCE node
I20260810 23:03:45.790354 32401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:45.790550 32401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:45.791688 32401 hybrid_clock.cc:648] HybridClock initialized: now 1786403025791667 us; error 42 us; skew 500 ppm
Aug 10 23:03:45 dist-test-slave-1xrh krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:45.794274 32401 init.cc:377] Logged in from keytab as oryx/127.23.56.66@KRBTEST.COM (short username oryx)
I20260810 23:03:45.794781 32401 webserver.cc:460] Webserver started at http://127.23.56.66:42703/ using document root <none> and password file <none>
I20260810 23:03:45.795045 32401 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:45.795114 32401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:45.795292 32401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:45.796304 32401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "b3dbd906239f4875be13f7cfd0e56386"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "4527583b5072356c259e3051496c9a32"
server_key_iv: "21088c829addf101a49a427d3db59227"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.796672 32401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "b3dbd906239f4875be13f7cfd0e56386"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "4527583b5072356c259e3051496c9a32"
server_key_iv: "21088c829addf101a49a427d3db59227"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.797966 32401 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:45.798744 32421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:45.798949 32401 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:45.799058 32401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "b3dbd906239f4875be13f7cfd0e56386"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "4527583b5072356c259e3051496c9a32"
server_key_iv: "21088c829addf101a49a427d3db59227"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.799170 32401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:45.824878 32401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:45.826817 32401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:45.826982 32401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:45.827641 32401 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:45.827704 32401 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:45.827749 32401 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:45.827785 32401 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:03:45 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:45.835233 32401 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:33227
I20260810 23:03:45.835312 32560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:33227 every 8 connection(s)
I20260810 23:03:45.835651 32401 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:45.837325 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.827997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:37341 (local address 127.23.56.126:46733)
0810 23:03:45.828056 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:45.828059 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:45.828317 (+ 258us) server_negotiation.cc:407] Connection header received
0810 23:03:45.828811 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:45.828813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:45.828828 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:45.828918 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:45.829546 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.830383 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.831672 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.831887 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.831971 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.834536 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:45.834538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:45.834538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:45.834546 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:45.836221 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.836639 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.836640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.836640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.836693 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.836875 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.836876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.836876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.836973 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:45.837024 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:45.837206 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 23:03:45.837263 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:03:45.837527 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.828014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46733 (local address 127.23.56.66:37341)
0810 23:03:45.828181 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:45.828223 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:45.828713 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:45.828965 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:45.828968 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:45.829198 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:45.829434 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.829439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.830475 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.830476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.831580 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.831582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.831995 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.831996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.832122 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.832631 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:45.832643 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:45.834436 (+ 1793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:45.836289 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.836292 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.836294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.836572 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.836736 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.836736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.836737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.836826 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.837014 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:45.837017 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:45.837116 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 23:03:45.837193 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:03:45.837313 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260810 23:03:45.838032 32562 heartbeater.cc:347] Connected to a master server at 127.23.56.126:46733
I20260810 23:03:45.838120 32562 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:45.838316 32562 heartbeater.cc:511] Master 127.23.56.126:46733 requested a full tablet report, sending...
I20260810 23:03:45.838737 32142 ts_manager.cc:194] Registered new tserver with Master: b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227)
I20260810 23:03:45.839215 32142 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:37341
I20260810 23:03:45.844271 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 32401
I20260810 23:03:45.844421 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:45.844499 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.839280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:50497 (local address 127.23.56.126:46733)
0810 23:03:45.839320 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:45.839322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:45.839437 (+ 115us) server_negotiation.cc:407] Connection header received
0810 23:03:45.839528 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:45.839529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:45.839538 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:45.839586 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:45.839915 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.840368 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.841055 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.841262 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.841337 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.842111 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:45.842113 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:45.842113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:45.842120 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:45.843331 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.843661 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.843662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.843663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.843715 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.843938 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.843939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.843939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.844104 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:45.844183 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:45.844336 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:03:45.844410 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 23:03:45.844535 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.839210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46733 (local address 127.23.56.66:50497)
0810 23:03:45.839381 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:45.839391 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:45.839459 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:45.839589 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:45.839590 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:45.839773 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:45.839830 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.839832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.840464 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.840465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.840971 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.840973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.841336 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.841337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.841402 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.841763 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:45.841767 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:45.842022 (+ 255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:45.843418 (+ 1396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.843418 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.843419 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.843590 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.843803 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.843804 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.843804 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.843868 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.844174 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:45.844175 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:45.844221 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 23:03:45.844310 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 23:03:45.844410 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
I20260810 23:03:45.844667 23777 external_mini_cluster.cc:1428] Setting key 6f0d72117a581f460fb41a7b6346b018
WARNING: no policy specified for oryx/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab.
Entry for principal oryx/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab.
I20260810 23:03:45.870409 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:46733
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab
--principal=oryx/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:45.950975 32573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:45.951203 32573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:45.951226 32573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:45.952384 32573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:45.952422 32573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:45.952442 32573 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 23:03:45.952461 32573 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:45.953985 32573 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab
--principal=oryx/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.56.126:46733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:45.954244 32573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:45.954500 32573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:45.959184 32582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:45.959235 32578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:45.959192 32579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:45.959524 32573 server_base.cc:1034] running on GCE node
I20260810 23:03:45.959774 32573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:45.959970 32573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:45.961162 32573 hybrid_clock.cc:648] HybridClock initialized: now 1786403025961153 us; error 33 us; skew 500 ppm
Aug 10 23:03:45 dist-test-slave-1xrh krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:45.963841 32573 init.cc:377] Logged in from keytab as oryx/127.23.56.67@KRBTEST.COM (short username oryx)
I20260810 23:03:45.964326 32573 webserver.cc:460] Webserver started at http://127.23.56.67:43777/ using document root <none> and password file <none>
I20260810 23:03:45.964659 32573 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:45.964758 32573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:45.964932 32573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:45.966043 32573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "03392a66d2f947c1b966bd13a1efde4f"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "401b441cf7d9d552f68145f68bd2b57b"
server_key_iv: "bbc9b3d8de159ef4186b1e18822deef4"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.966392 32573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "03392a66d2f947c1b966bd13a1efde4f"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "401b441cf7d9d552f68145f68bd2b57b"
server_key_iv: "bbc9b3d8de159ef4186b1e18822deef4"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.967794 32573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:03:45.968520 32589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:45.968760 32573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:45.968854 32573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "03392a66d2f947c1b966bd13a1efde4f"
format_stamp: "Formatted at 2026-08-10 23:03:45 on dist-test-slave-1xrh"
server_key: "401b441cf7d9d552f68145f68bd2b57b"
server_key_iv: "bbc9b3d8de159ef4186b1e18822deef4"
server_key_version: "encryptionkey@0"
I20260810 23:03:45.968942 32573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:45.980672 32573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:45.982496 32573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:45.982656 32573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:45.983292 32573 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:45.983353 32573 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:45.983404 32573 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:45.983433 32573 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 23:03:45 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:45.991113 32573 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:46307
I20260810 23:03:45.991272 32727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:46307 every 8 connection(s)
I20260810 23:03:45.991537 32573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:45.993744 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.983663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:39225 (local address 127.23.56.126:46733)
0810 23:03:45.983740 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:45.983743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:45.984035 (+ 292us) server_negotiation.cc:407] Connection header received
0810 23:03:45.984775 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:45.984776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:45.984791 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:45.984894 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:45.985780 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.986398 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.988104 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.988310 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.988391 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.991174 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:45.991176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:45.991176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:45.991186 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:45.992663 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.993037 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.993038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.993038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.993088 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:45.993277 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:45.993278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:45.993278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:45.993375 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:45.993423 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:45.993620 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 23:03:45.993683 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:03:45.993891 32599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.983663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46733 (local address 127.23.56.67:39225)
0810 23:03:45.983871 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:45.983925 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:45.984633 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:45.984943 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:45.984948 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:45.985362 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:45.985678 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.985684 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.986475 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.986476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.987990 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.987993 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.988409 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.988411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.988541 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.989080 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:45.989090 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:45.991039 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:45.992749 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.992755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.992757 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.992965 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.993144 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:45.993144 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:45.993145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:45.993228 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:45.993412 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:45.993415 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:45.993517 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 23:03:45.993591 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:03:45.993704 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260810 23:03:45.994462 32728 heartbeater.cc:347] Connected to a master server at 127.23.56.126:46733
I20260810 23:03:45.994549 32728 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:45.994716 32728 heartbeater.cc:511] Master 127.23.56.126:46733 requested a full tablet report, sending...
I20260810 23:03:45.995072 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 32573
I20260810 23:03:45.995175 32143 ts_manager.cc:194] Registered new tserver with Master: 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307)
I20260810 23:03:45.995215 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:45.995436 23777 external_mini_cluster.cc:1428] Setting key 6a316e36ddf3ff78dcab6fdca1f89f51
I20260810 23:03:45.995757 32143 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:39225
I20260810 23:03:45.996356 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:03:46.001924 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.995583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:37781 (local address 127.23.56.126:46733)
0810 23:03:45.995751 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:45.995753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:45.995870 (+ 117us) server_negotiation.cc:407] Connection header received
0810 23:03:45.995979 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:45.995995 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:45.996006 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:45.996037 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:45.996426 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.997099 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.997757 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.998061 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.998132 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.999176 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:45.999178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:45.999178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:45.999185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:46.000575 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:46.000997 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:46.000998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:46.000998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:46.001072 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:46.001360 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:46.001361 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:46.001361 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:46.001460 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:46.001520 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.001751 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.001833 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260810 23:03:46.001963 32736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:45.995509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46733 (local address 127.23.56.67:37781)
0810 23:03:45.995815 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:45.995825 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:45.995893 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:45.996095 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:45.996095 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:45.996290 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:45.996355 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.996358 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.997197 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.997197 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.997666 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:45.997667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:45.998139 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:45.998140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:45.998186 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:45.998622 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:45.998628 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:45.999044 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:46.000656 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:46.000657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:46.000658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:46.000901 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:46.001152 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:46.001153 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:46.001154 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:46.001227 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:46.001531 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:46.001532 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:46.001585 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.001716 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.001848 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":77,"threads_started":1}
Aug 10 23:03:46 dist-test-slave-1xrh krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 23:03:46 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:46 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:03:46.019624 32205 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:46.014428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.23.56.126:46733)
0810 23:03:46.014494 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.014496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.014506 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:46.014554 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.014555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.014564 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.014612 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:46.015253 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.015720 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.016424 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.016594 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.016642 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.019507 (+ 2865us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52434: BlockingRecv error: recv got EOF from 127.0.0.1:52434 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 23:03:46 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403026, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:46.026427 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.020275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52446 (local address 127.23.56.126:46733)
0810 23:03:46.020342 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.020344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.020394 (+ 50us) server_negotiation.cc:407] Connection header received
0810 23:03:46.020423 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.020424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.020432 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.020468 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:46.020911 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.021326 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.021929 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.022092 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.022138 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.024132 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:46.024133 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:46.024133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:46.024139 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:46.025509 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:46.025801 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:46.025802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:46.025802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:46.025849 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:46.026032 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:46.026033 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:46.026033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:46.026122 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:46.026193 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.026271 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.026327 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:03:46.027907 32143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:46.029417 32143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:46.035715 32754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.031589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:33227 (local address 127.0.0.1:57872)
0810 23:03:46.031730 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.031773 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.031834 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.032333 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.032336 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.032338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:46.032405 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.032407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.033297 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.033298 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.034356 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.034359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.035393 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.035395 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.035482 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.035498 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.035572 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.035626 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":44,"threads_started":1}
I20260810 23:03:46.035753 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.031704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57872 (local address 127.23.56.66:33227)
0810 23:03:46.032091 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.032096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.032108 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:03:46.032149 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.032151 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.032167 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.032249 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:46.032489 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.033211 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.034443 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.035274 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.035380 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.035458 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.035598 (+ 140us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.035642 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":93,"threads_started":1}
I20260810 23:03:46.036029 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.031841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43878 (local address 127.23.56.67:46307)
0810 23:03:46.032020 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.032024 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.032291 (+ 267us) server_negotiation.cc:407] Connection header received
0810 23:03:46.032337 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.032339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.032353 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.032421 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:46.032625 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.033629 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.034796 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.035418 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.035494 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.035569 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.035887 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.035938 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":95,"threads_started":1}
I20260810 23:03:46.036113 32756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.031615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:33051 (local address 127.0.0.1:52904)
0810 23:03:46.032019 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.032030 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.032085 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.032342 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.032343 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.032345 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:46.032418 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.032421 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.033697 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.033698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.034721 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.034724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.035804 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.035805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.035928 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.035931 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.036002 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.036040 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":39,"thread_start_us":95,"threads_started":1}
I20260810 23:03:46.036146 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.031856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.23.56.65:33051)
0810 23:03:46.032090 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.032093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.032111 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:03:46.032144 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.032145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.032158 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.032249 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:46.032520 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.033583 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.034826 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.035635 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.035700 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.035767 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.036019 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.036052 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260810 23:03:46.035820 32755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.031624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:46307 (local address 127.0.0.1:43878)
0810 23:03:46.032180 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.032189 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.032234 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.032503 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.032504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.032506 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:46.032542 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.032544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.033711 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.033712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.034702 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.034704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.035605 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.035606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.035656 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.035660 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.035707 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.035743 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":9,"thread_start_us":84,"threads_started":1}
I20260810 23:03:46.036900 32311 tablet_service.cc:1467] Processing CreateTablet for tablet 6224cb9e2ff64bda93538b5c9bce46f9 (DEFAULT_TABLE table=test-table [id=eee1eb47894140d0933d4b4145ea43d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.037079 32647 tablet_service.cc:1467] Processing CreateTablet for tablet 6224cb9e2ff64bda93538b5c9bce46f9 (DEFAULT_TABLE table=test-table [id=eee1eb47894140d0933d4b4145ea43d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.037274 32311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6224cb9e2ff64bda93538b5c9bce46f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.037204 32481 tablet_service.cc:1467] Processing CreateTablet for tablet 6224cb9e2ff64bda93538b5c9bce46f9 (DEFAULT_TABLE table=test-table [id=eee1eb47894140d0933d4b4145ea43d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.037425 32647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6224cb9e2ff64bda93538b5c9bce46f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.037488 32481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6224cb9e2ff64bda93538b5c9bce46f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.040786 32760 tablet_bootstrap.cc:492] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: Bootstrap starting.
I20260810 23:03:46.041181 32761 tablet_bootstrap.cc:492] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: Bootstrap starting.
I20260810 23:03:46.041433 32763 tablet_bootstrap.cc:492] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: Bootstrap starting.
I20260810 23:03:46.041564 32760 tablet_bootstrap.cc:654] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.041798 32761 tablet_bootstrap.cc:654] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.041903 32760 log.cc:824] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:46.042062 32763 tablet_bootstrap.cc:654] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.042119 32761 log.cc:824] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:46.042367 32763 log.cc:824] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:46.042922 32761 tablet_bootstrap.cc:492] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: No bootstrap required, opened a new log
I20260810 23:03:46.042986 32761 ts_tablet_manager.cc:1397] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:03:46.043200 32763 tablet_bootstrap.cc:492] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: No bootstrap required, opened a new log
I20260810 23:03:46.043288 32763 ts_tablet_manager.cc:1397] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:46.043493 32760 tablet_bootstrap.cc:492] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: No bootstrap required, opened a new log
I20260810 23:03:46.043562 32760 ts_tablet_manager.cc:1397] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:03:46.044688 32763 raft_consensus.cc:348] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.044840 32763 raft_consensus.cc:374] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.044885 32763 raft_consensus.cc:729] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3dbd906239f4875be13f7cfd0e56386, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.045017 32763 consensus_queue.cc:260] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.045022 32761 raft_consensus.cc:348] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.045171 32761 raft_consensus.cc:374] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.045203 32761 raft_consensus.cc:729] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03392a66d2f947c1b966bd13a1efde4f, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.045292 32763 ts_tablet_manager.cc:1428] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:03:46.045320 32761 consensus_queue.cc:260] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.045531 32562 heartbeater.cc:503] Master 127.23.56.126:46733 was elected leader, sending a full tablet report...
I20260810 23:03:46.045563 32761 ts_tablet_manager.cc:1428] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:46.045666 32728 heartbeater.cc:503] Master 127.23.56.126:46733 was elected leader, sending a full tablet report...
I20260810 23:03:46.045562 32760 raft_consensus.cc:348] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.045974 32760 raft_consensus.cc:374] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.046031 32760 raft_consensus.cc:729] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571a317f77bd45c5bd9b66fb7a478503, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.046136 32760 consensus_queue.cc:260] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.046397 32760 ts_tablet_manager.cc:1428] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:03:46.046469 32390 heartbeater.cc:503] Master 127.23.56.126:46733 was elected leader, sending a full tablet report...
I20260810 23:03:46.059538 300 raft_consensus.cc:482] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:46.059646 300 raft_consensus.cc:504] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.059950 300 leader_election.cc:288] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051), 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307)
I20260810 23:03:46.063695 32758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.060233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:36431 (local address 127.23.56.67:46307)
0810 23:03:46.060272 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.060274 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.060389 (+ 115us) server_negotiation.cc:407] Connection header received
0810 23:03:46.060443 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.060444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.060453 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.060508 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:46.060649 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.061321 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.062788 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.063337 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.063415 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.063423 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.063597 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.063633 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:03:46.063695 32759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.060219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:49123 (local address 127.23.56.65:33051)
0810 23:03:46.060284 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.060286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.060294 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:03:46.060323 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.060324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.060331 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.060410 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:46.060566 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.061321 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.062788 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.063350 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.063415 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.063423 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.063597 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.063634 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:03:46.063758 305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.060130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:46307 (local address 127.23.56.66:36431)
0810 23:03:46.060326 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.060333 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.060395 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.060510 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.060511 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.060513 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:46.060581 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.060584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.061404 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.061405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.062695 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.062699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.063426 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.063427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.063513 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.063518 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.063578 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.063617 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":87,"thread_start_us":82,"threads_started":1}
I20260810 23:03:46.063768 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.060130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:33051 (local address 127.23.56.66:49123)
0810 23:03:46.060172 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.060184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.060227 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.060413 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.060414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.060416 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:46.060484 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.060486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.061404 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.061405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.062695 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.062699 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.063426 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.063427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.063520 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.063523 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.063578 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.063617 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":84}
I20260810 23:03:46.064081 32674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6224cb9e2ff64bda93538b5c9bce46f9" candidate_uuid: "b3dbd906239f4875be13f7cfd0e56386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03392a66d2f947c1b966bd13a1efde4f" is_pre_election: true
I20260810 23:03:46.064219 32674 raft_consensus.cc:2454] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3dbd906239f4875be13f7cfd0e56386 in term 0.
I20260810 23:03:46.064321 32333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6224cb9e2ff64bda93538b5c9bce46f9" candidate_uuid: "b3dbd906239f4875be13f7cfd0e56386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571a317f77bd45c5bd9b66fb7a478503" is_pre_election: true
I20260810 23:03:46.064445 32333 raft_consensus.cc:2454] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3dbd906239f4875be13f7cfd0e56386 in term 0.
I20260810 23:03:46.064455 32427 leader_election.cc:302] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03392a66d2f947c1b966bd13a1efde4f, b3dbd906239f4875be13f7cfd0e56386; no voters:
I20260810 23:03:46.064565 300 raft_consensus.cc:2781] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:46.064633 300 raft_consensus.cc:482] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:46.064667 300 raft_consensus.cc:3037] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.065548 300 raft_consensus.cc:504] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.065704 300 leader_election.cc:288] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [CANDIDATE]: Term 1 election: Requested vote from peers 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051), 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307)
I20260810 23:03:46.065886 32674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6224cb9e2ff64bda93538b5c9bce46f9" candidate_uuid: "b3dbd906239f4875be13f7cfd0e56386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03392a66d2f947c1b966bd13a1efde4f"
I20260810 23:03:46.065886 32333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6224cb9e2ff64bda93538b5c9bce46f9" candidate_uuid: "b3dbd906239f4875be13f7cfd0e56386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571a317f77bd45c5bd9b66fb7a478503"
I20260810 23:03:46.065961 32333 raft_consensus.cc:3037] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.065969 32674 raft_consensus.cc:3037] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.066613 32333 raft_consensus.cc:2454] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3dbd906239f4875be13f7cfd0e56386 in term 1.
I20260810 23:03:46.066721 32674 raft_consensus.cc:2454] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3dbd906239f4875be13f7cfd0e56386 in term 1.
I20260810 23:03:46.066792 32426 leader_election.cc:302] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 571a317f77bd45c5bd9b66fb7a478503, b3dbd906239f4875be13f7cfd0e56386; no voters:
I20260810 23:03:46.066877 300 raft_consensus.cc:2781] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:46.067013 300 raft_consensus.cc:686] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 1 LEADER]: Becoming Leader. State: Replica: b3dbd906239f4875be13f7cfd0e56386, State: Running, Role: LEADER
I20260810 23:03:46.067162 300 consensus_queue.cc:237] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.067984 32143 catalog_manager.cc:5495] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66). New cstate: current_term: 1 leader_uuid: "b3dbd906239f4875be13f7cfd0e56386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260810 23:03:46.073148 32757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.070716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57888 (local address 127.23.56.66:33227)
0810 23:03:46.070760 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.070762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.070771 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:46.070843 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.070844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.070860 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.070923 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:46.071066 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.071626 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.072379 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.072593 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.072670 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.072839 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:46.072930 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:46.073033 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.073059 (+ 26us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.073079 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:03:46.074723 32333 raft_consensus.cc:1261] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Refusing update from remote peer b3dbd906239f4875be13f7cfd0e56386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:46.075017 32674 raft_consensus.cc:1261] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Refusing update from remote peer b3dbd906239f4875be13f7cfd0e56386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:46.075031 306 consensus_queue.cc:1035] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:46.075366 306 consensus_queue.cc:1035] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:46.077230 310 mvcc.cc:205] Tried to move back new op lower bound from 7317106794799595520 to 7317106794771918848. Current Snapshot: MvccSnapshot[applied={T|T < 7317106794799595520}]
I20260810 23:03:46.077401 311 mvcc.cc:205] Tried to move back new op lower bound from 7317106794799595520 to 7317106794771918848. Current Snapshot: MvccSnapshot[applied={T|T < 7317106794799595520}]
I20260810 23:03:46.079725 32143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:46.079797 32143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:46.080528 32143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c: Sending DeleteTablet for 3 replicas of tablet 6224cb9e2ff64bda93538b5c9bce46f9
I20260810 23:03:46.080859 32311 tablet_service.cc:1514] Processing DeleteTablet for tablet 6224cb9e2ff64bda93538b5c9bce46f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:46 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:52904
I20260810 23:03:46.080868 32481 tablet_service.cc:1514] Processing DeleteTablet for tablet 6224cb9e2ff64bda93538b5c9bce46f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:46 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:57872
I20260810 23:03:46.080875 32647 tablet_service.cc:1514] Processing DeleteTablet for tablet 6224cb9e2ff64bda93538b5c9bce46f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:46 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:43878
I20260810 23:03:46.081097 320 tablet_replica.cc:330] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: stopping tablet replica
I20260810 23:03:46.081192 320 raft_consensus.cc:2229] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:46.081281 320 raft_consensus.cc:2258] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:46.081470 32143 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:46.081633 32143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:46.081733 320 ts_tablet_manager.cc:1905] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:46.082054 319 tablet_replica.cc:330] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: stopping tablet replica
I20260810 23:03:46.082566 319 raft_consensus.cc:2229] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:46.082748 319 raft_consensus.cc:2258] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:46.082827 32481 tablet_service.cc:1467] Processing CreateTablet for tablet 26cec247893a4ade983b513bda696b11 (DEFAULT_TABLE table=test-table [id=64e5f36b020644ff86028608fa41132f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.082943 32481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cec247893a4ade983b513bda696b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.083333 319 ts_tablet_manager.cc:1905] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:46.083511 320 ts_tablet_manager.cc:1918] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:46.083590 320 log.cc:1196] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/6224cb9e2ff64bda93538b5c9bce46f9
I20260810 23:03:46.083889 320 ts_tablet_manager.cc:1939] T 6224cb9e2ff64bda93538b5c9bce46f9 P b3dbd906239f4875be13f7cfd0e56386: Deleting consensus metadata
I20260810 23:03:46.084260 32763 tablet_bootstrap.cc:492] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: Bootstrap starting.
I20260810 23:03:46.084509 32647 tablet_service.cc:1467] Processing CreateTablet for tablet 26cec247893a4ade983b513bda696b11 (DEFAULT_TABLE table=test-table [id=64e5f36b020644ff86028608fa41132f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.084610 32647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cec247893a4ade983b513bda696b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.084682 32763 tablet_bootstrap.cc:654] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.084738 32311 tablet_service.cc:1467] Processing CreateTablet for tablet 26cec247893a4ade983b513bda696b11 (DEFAULT_TABLE table=test-table [id=64e5f36b020644ff86028608fa41132f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.084844 32311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26cec247893a4ade983b513bda696b11. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.085227 32125 catalog_manager.cc:4841] TS b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227): tablet 6224cb9e2ff64bda93538b5c9bce46f9 (table test-table [id=eee1eb47894140d0933d4b4145ea43d3]) successfully deleted
I20260810 23:03:46.085623 32763 tablet_bootstrap.cc:492] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: No bootstrap required, opened a new log
I20260810 23:03:46.085661 32763 ts_tablet_manager.cc:1397] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:46.085788 32763 raft_consensus.cc:348] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.085860 32763 raft_consensus.cc:374] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.085880 32763 raft_consensus.cc:729] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3dbd906239f4875be13f7cfd0e56386, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.085927 32761 tablet_bootstrap.cc:492] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: Bootstrap starting.
I20260810 23:03:46.085923 32763 consensus_queue.cc:260] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.086020 32763 ts_tablet_manager.cc:1428] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260810 23:03:46.086272 32563 tablet.cc:2369] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:03:46.086344 32761 tablet_bootstrap.cc:654] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.086385 32760 tablet_bootstrap.cc:492] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: Bootstrap starting.
I20260810 23:03:46.086804 32760 tablet_bootstrap.cc:654] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.086966 32761 tablet_bootstrap.cc:492] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: No bootstrap required, opened a new log
I20260810 23:03:46.087029 32761 ts_tablet_manager.cc:1397] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:46.087189 32761 raft_consensus.cc:348] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.087251 32761 raft_consensus.cc:374] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.087273 32761 raft_consensus.cc:729] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03392a66d2f947c1b966bd13a1efde4f, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.087329 32761 consensus_queue.cc:260] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.087424 32761 ts_tablet_manager.cc:1428] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:46.089177 319 ts_tablet_manager.cc:1918] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:46.089277 319 log.cc:1196] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/6224cb9e2ff64bda93538b5c9bce46f9
I20260810 23:03:46.089557 319 ts_tablet_manager.cc:1939] T 6224cb9e2ff64bda93538b5c9bce46f9 P 03392a66d2f947c1b966bd13a1efde4f: Deleting consensus metadata
I20260810 23:03:46.089927 32127 catalog_manager.cc:4841] TS 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307): tablet 6224cb9e2ff64bda93538b5c9bce46f9 (table test-table [id=eee1eb47894140d0933d4b4145ea43d3]) successfully deleted
I20260810 23:03:46.091243 321 tablet_replica.cc:330] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: stopping tablet replica
I20260810 23:03:46.091360 321 raft_consensus.cc:2229] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:46.091481 321 raft_consensus.cc:2258] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:46.091904 321 ts_tablet_manager.cc:1905] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:46.092006 32760 tablet_bootstrap.cc:492] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: No bootstrap required, opened a new log
I20260810 23:03:46.092051 32760 ts_tablet_manager.cc:1397] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 23:03:46.092183 32760 raft_consensus.cc:348] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.092242 32760 raft_consensus.cc:374] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.092262 32760 raft_consensus.cc:729] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571a317f77bd45c5bd9b66fb7a478503, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.092303 32760 consensus_queue.cc:260] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.092382 32760 ts_tablet_manager.cc:1428] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:46.093500 321 ts_tablet_manager.cc:1918] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:46.093559 321 log.cc:1196] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/6224cb9e2ff64bda93538b5c9bce46f9
I20260810 23:03:46.093734 321 ts_tablet_manager.cc:1939] T 6224cb9e2ff64bda93538b5c9bce46f9 P 571a317f77bd45c5bd9b66fb7a478503: Deleting consensus metadata
I20260810 23:03:46.094251 32126 catalog_manager.cc:4841] TS 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051): tablet 6224cb9e2ff64bda93538b5c9bce46f9 (table test-table [id=eee1eb47894140d0933d4b4145ea43d3]) successfully deleted
W20260810 23:03:46.168738 32391 tablet.cc:2369] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:03:46.242161 32730 tablet.cc:2369] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:03:46.389286 306 raft_consensus.cc:482] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:46.389380 306 raft_consensus.cc:504] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.389544 306 leader_election.cc:288] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051), 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307)
I20260810 23:03:46.389791 32674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cec247893a4ade983b513bda696b11" candidate_uuid: "b3dbd906239f4875be13f7cfd0e56386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03392a66d2f947c1b966bd13a1efde4f" is_pre_election: true
I20260810 23:03:46.389818 32333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cec247893a4ade983b513bda696b11" candidate_uuid: "b3dbd906239f4875be13f7cfd0e56386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571a317f77bd45c5bd9b66fb7a478503" is_pre_election: true
I20260810 23:03:46.389907 32674 raft_consensus.cc:2454] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3dbd906239f4875be13f7cfd0e56386 in term 0.
I20260810 23:03:46.389935 32333 raft_consensus.cc:2454] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3dbd906239f4875be13f7cfd0e56386 in term 0.
I20260810 23:03:46.390087 32427 leader_election.cc:302] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03392a66d2f947c1b966bd13a1efde4f, b3dbd906239f4875be13f7cfd0e56386; no voters:
I20260810 23:03:46.390208 306 raft_consensus.cc:2781] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:46.390260 306 raft_consensus.cc:482] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:46.390290 306 raft_consensus.cc:3037] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.391022 306 raft_consensus.cc:504] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.391191 306 leader_election.cc:288] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [CANDIDATE]: Term 1 election: Requested vote from peers 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051), 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307)
I20260810 23:03:46.391358 32674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cec247893a4ade983b513bda696b11" candidate_uuid: "b3dbd906239f4875be13f7cfd0e56386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03392a66d2f947c1b966bd13a1efde4f"
I20260810 23:03:46.391358 32333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26cec247893a4ade983b513bda696b11" candidate_uuid: "b3dbd906239f4875be13f7cfd0e56386" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571a317f77bd45c5bd9b66fb7a478503"
I20260810 23:03:46.391429 32674 raft_consensus.cc:3037] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.391430 32333 raft_consensus.cc:3037] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.392184 32333 raft_consensus.cc:2454] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3dbd906239f4875be13f7cfd0e56386 in term 1.
I20260810 23:03:46.392184 32674 raft_consensus.cc:2454] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3dbd906239f4875be13f7cfd0e56386 in term 1.
I20260810 23:03:46.392352 32427 leader_election.cc:302] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03392a66d2f947c1b966bd13a1efde4f, b3dbd906239f4875be13f7cfd0e56386; no voters:
I20260810 23:03:46.392446 306 raft_consensus.cc:2781] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:46.392498 306 raft_consensus.cc:686] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 1 LEADER]: Becoming Leader. State: Replica: b3dbd906239f4875be13f7cfd0e56386, State: Running, Role: LEADER
I20260810 23:03:46.392547 306 consensus_queue.cc:237] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.393144 32143 catalog_manager.cc:5495] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66). New cstate: current_term: 1 leader_uuid: "b3dbd906239f4875be13f7cfd0e56386" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } health_report { overall_health: UNKNOWN } } }
Aug 10 23:03:46 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403025, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.126@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:46.472672 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.463905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46733 (local address 127.0.0.1:52452)
0810 23:03:46.464081 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.464090 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.464155 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.464278 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.464279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.464589 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:46.464655 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.464658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.465464 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.465465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.466208 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.466211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.466691 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.466692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.466749 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.467368 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:46.467374 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:46.469278 (+ 1904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:46.471374 (+ 2096us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:46.471382 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:46.471385 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:46.471646 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:46.471881 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:46.471881 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:46.471882 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:46.471950 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:46.472269 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:46.472276 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:46.472331 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.472421 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.472534 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260810 23:03:46.472764 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.463971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52452 (local address 127.23.56.126:46733)
0810 23:03:46.464075 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.464078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.464140 (+ 62us) server_negotiation.cc:407] Connection header received
0810 23:03:46.464210 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.464211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.464220 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.464261 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:46.464728 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.465368 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.466314 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.466625 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.466690 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.469363 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:46.469365 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:46.469365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:46.469373 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:46.471291 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:46.471727 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:46.471729 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:46.471729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:46.471800 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:46.472036 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:46.472037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:46.472037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:46.472181 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:46.472255 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.472534 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.472631 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 23:03:46.476735 32143 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:52452:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 23:03:46.476996 32143 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:46.478618 32311 tablet_service.cc:1467] Processing CreateTablet for tablet 5d4f6a568b2440c484f7340612d71a3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea11963358434a16bbfab91d2cbf92e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:03:46.478621 32647 tablet_service.cc:1467] Processing CreateTablet for tablet 5d4f6a568b2440c484f7340612d71a3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea11963358434a16bbfab91d2cbf92e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:03:46.478799 32311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d4f6a568b2440c484f7340612d71a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.478811 32647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d4f6a568b2440c484f7340612d71a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.480031 32760 tablet_bootstrap.cc:492] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: Bootstrap starting.
I20260810 23:03:46.480060 32761 tablet_bootstrap.cc:492] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: Bootstrap starting.
I20260810 23:03:46.480166 32481 tablet_service.cc:1467] Processing CreateTablet for tablet 5d4f6a568b2440c484f7340612d71a3f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ea11963358434a16bbfab91d2cbf92e7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 23:03:46.480325 32481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5d4f6a568b2440c484f7340612d71a3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.480468 32760 tablet_bootstrap.cc:654] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.480485 32761 tablet_bootstrap.cc:654] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.481130 32761 tablet_bootstrap.cc:492] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: No bootstrap required, opened a new log
I20260810 23:03:46.481194 32761 ts_tablet_manager.cc:1397] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:46.481338 32761 raft_consensus.cc:348] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.481411 32761 raft_consensus.cc:374] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.481429 32761 raft_consensus.cc:729] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03392a66d2f947c1b966bd13a1efde4f, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.481470 32761 consensus_queue.cc:260] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.481575 32761 ts_tablet_manager.cc:1428] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:46.481765 302 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.481863 302 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:46.481983 32763 tablet_bootstrap.cc:492] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: Bootstrap starting.
I20260810 23:03:46.482398 32763 tablet_bootstrap.cc:654] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.483144 32763 tablet_bootstrap.cc:492] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: No bootstrap required, opened a new log
I20260810 23:03:46.483227 32763 ts_tablet_manager.cc:1397] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:46.483341 32763 raft_consensus.cc:348] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.483408 32763 raft_consensus.cc:374] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.483428 32763 raft_consensus.cc:729] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3dbd906239f4875be13f7cfd0e56386, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.483489 32763 consensus_queue.cc:260] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.483584 32763 ts_tablet_manager.cc:1428] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:46.483611 306 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.483711 306 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:46.485004 32760 tablet_bootstrap.cc:492] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: No bootstrap required, opened a new log
I20260810 23:03:46.485049 32760 ts_tablet_manager.cc:1397] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260810 23:03:46.485217 32760 raft_consensus.cc:348] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.485284 32760 raft_consensus.cc:374] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.485299 32760 raft_consensus.cc:729] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571a317f77bd45c5bd9b66fb7a478503, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.485351 32760 consensus_queue.cc:260] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.485458 32760 ts_tablet_manager.cc:1428] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:46.485503 304 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.485579 304 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:46.576073 302 raft_consensus.cc:482] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:46.576179 302 raft_consensus.cc:504] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.576596 302 leader_election.cc:288] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051), b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227)
I20260810 23:03:46.579879 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.576821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:45395 (local address 127.23.56.65:33051)
0810 23:03:46.576952 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.576954 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.576968 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:03:46.577057 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.577058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.577069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.577159 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:46.577342 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.577971 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.579001 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.579518 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.579578 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.579586 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.579765 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.579794 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":50,"threads_started":1}
I20260810 23:03:46.579885 337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.576765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:33051 (local address 127.23.56.67:45395)
0810 23:03:46.576910 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.576921 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.576981 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.577176 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.577177 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.577179 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:46.577243 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.577244 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.578051 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.578052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.578910 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.578914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.579600 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.579601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.579678 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.579683 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.579744 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.579788 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":61,"threads_started":1}
I20260810 23:03:46.580156 338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.576751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:33227 (local address 127.23.56.67:50067)
0810 23:03:46.576883 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.576894 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.576950 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.577140 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.577141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.577144 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:46.577229 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.577231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.578080 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.578081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.579007 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.579009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.579881 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.579882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.579957 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.579963 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.580028 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.580075 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":20,"thread_start_us":63,"threads_started":1}
I20260810 23:03:46.580152 32333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d4f6a568b2440c484f7340612d71a3f" candidate_uuid: "03392a66d2f947c1b966bd13a1efde4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571a317f77bd45c5bd9b66fb7a478503" is_pre_election: true
I20260810 23:03:46.580235 32333 raft_consensus.cc:2454] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03392a66d2f947c1b966bd13a1efde4f in term 0.
I20260810 23:03:46.580399 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.576825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:50067 (local address 127.23.56.66:33227)
0810 23:03:46.576943 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.576945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.576955 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:46.577047 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.577049 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.577066 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.577117 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:46.577299 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.577960 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.579107 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.579753 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.579876 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.579885 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.580271 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.580327 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":48,"threads_started":1}
I20260810 23:03:46.580402 32593 leader_election.cc:302] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03392a66d2f947c1b966bd13a1efde4f, 571a317f77bd45c5bd9b66fb7a478503; no voters:
I20260810 23:03:46.580557 302 raft_consensus.cc:2781] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:46.580602 302 raft_consensus.cc:482] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:46.580621 302 raft_consensus.cc:3037] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.580619 32507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d4f6a568b2440c484f7340612d71a3f" candidate_uuid: "03392a66d2f947c1b966bd13a1efde4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dbd906239f4875be13f7cfd0e56386" is_pre_election: true
I20260810 23:03:46.580777 32507 raft_consensus.cc:2454] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03392a66d2f947c1b966bd13a1efde4f in term 0.
I20260810 23:03:46.581393 302 raft_consensus.cc:504] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.581496 302 leader_election.cc:288] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [CANDIDATE]: Term 1 election: Requested vote from peers 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051), b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227)
I20260810 23:03:46.581679 32507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d4f6a568b2440c484f7340612d71a3f" candidate_uuid: "03392a66d2f947c1b966bd13a1efde4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dbd906239f4875be13f7cfd0e56386"
I20260810 23:03:46.581681 32333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5d4f6a568b2440c484f7340612d71a3f" candidate_uuid: "03392a66d2f947c1b966bd13a1efde4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571a317f77bd45c5bd9b66fb7a478503"
I20260810 23:03:46.581754 32507 raft_consensus.cc:3037] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.581754 32333 raft_consensus.cc:3037] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.582533 32507 raft_consensus.cc:2454] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03392a66d2f947c1b966bd13a1efde4f in term 1.
I20260810 23:03:46.582530 32333 raft_consensus.cc:2454] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03392a66d2f947c1b966bd13a1efde4f in term 1.
I20260810 23:03:46.582705 32593 leader_election.cc:302] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03392a66d2f947c1b966bd13a1efde4f, 571a317f77bd45c5bd9b66fb7a478503; no voters:
I20260810 23:03:46.582805 302 raft_consensus.cc:2781] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:46.582906 302 raft_consensus.cc:686] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [term 1 LEADER]: Becoming Leader. State: Replica: 03392a66d2f947c1b966bd13a1efde4f, State: Running, Role: LEADER
I20260810 23:03:46.582973 302 consensus_queue.cc:237] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.583441 341 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: TxnStatusTablet state changed. Reason: New leader 03392a66d2f947c1b966bd13a1efde4f. Latest consensus state: current_term: 1 leader_uuid: "03392a66d2f947c1b966bd13a1efde4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.583560 341 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:03:46.583715 343 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 23:03:46.583854 343 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 23:03:46.583870 32143 catalog_manager.cc:5495] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67). New cstate: current_term: 1 leader_uuid: "03392a66d2f947c1b966bd13a1efde4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } health_report { overall_health: HEALTHY } } }
I20260810 23:03:46.600409 333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.597707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:46307 (local address 127.0.0.1:43894)
0810 23:03:46.597779 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.597791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.597859 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.598065 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.598066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.598068 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:46.598121 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.598124 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.598811 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.598812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.599486 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.599488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.599868 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.599869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.599941 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.599956 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:46.600234 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:46.600244 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.600289 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.600336 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 23:03:46.600406 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.597794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43894 (local address 127.23.56.67:46307)
0810 23:03:46.597915 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.597917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.597924 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:03:46.597952 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.597953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.597975 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.598058 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:46.598181 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.598726 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.599589 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.599793 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.599869 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.600066 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:46.600160 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:46.600236 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.600292 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.600311 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":43,"threads_started":1}
I20260810 23:03:46.601760 32333 raft_consensus.cc:1261] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Refusing update from remote peer 03392a66d2f947c1b966bd13a1efde4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:46.601910 32507 raft_consensus.cc:1261] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Refusing update from remote peer 03392a66d2f947c1b966bd13a1efde4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:46.602027 341 consensus_queue.cc:1035] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:46.602166 302 consensus_queue.cc:1035] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:46.602748 304 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: TxnStatusTablet state changed. Reason: New leader 03392a66d2f947c1b966bd13a1efde4f. Latest consensus state: current_term: 1 leader_uuid: "03392a66d2f947c1b966bd13a1efde4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.602836 304 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:46.602792 306 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: TxnStatusTablet state changed. Reason: New leader 03392a66d2f947c1b966bd13a1efde4f. Latest consensus state: current_term: 1 leader_uuid: "03392a66d2f947c1b966bd13a1efde4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.602902 306 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:46.603464 302 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03392a66d2f947c1b966bd13a1efde4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.603559 302 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:03:46.603826 302 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "03392a66d2f947c1b966bd13a1efde4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.603904 302 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P 03392a66d2f947c1b966bd13a1efde4f: This TxnStatusTablet replica's current role is: LEADER
I20260810 23:03:46.604110 306 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03392a66d2f947c1b966bd13a1efde4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.604192 306 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P b3dbd906239f4875be13f7cfd0e56386: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:46.604151 304 tablet_replica.cc:439] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "03392a66d2f947c1b966bd13a1efde4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } } }
I20260810 23:03:46.604224 304 tablet_replica.cc:442] T 5d4f6a568b2440c484f7340612d71a3f P 571a317f77bd45c5bd9b66fb7a478503: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 23:03:46.610534 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.608203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:37735 (local address 127.23.56.67:46307)
0810 23:03:46.608247 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.608249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.608346 (+ 97us) server_negotiation.cc:407] Connection header received
0810 23:03:46.608420 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.608421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.608432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.608480 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:46.608614 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.609062 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.609826 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.610027 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.610097 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.610295 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:46.610325 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:46.610395 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.610457 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.610475 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:03:46.610596 361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.608131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:46307 (local address 127.23.56.66:37735)
0810 23:03:46.608269 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.608280 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.608362 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.608493 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.608495 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.608497 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:46.608540 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.608541 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.609143 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.609144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.609731 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.609733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.610110 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.610111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.610191 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.610205 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:46.610395 (+ 190us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:46.610398 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.610440 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.610509 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":74,"threads_started":1}
I20260810 23:03:46.613066 32333 raft_consensus.cc:1261] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Refusing update from remote peer b3dbd906239f4875be13f7cfd0e56386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:46.613117 32674 raft_consensus.cc:1261] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Refusing update from remote peer b3dbd906239f4875be13f7cfd0e56386: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:46.613297 300 consensus_queue.cc:1035] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:46.613396 306 consensus_queue.cc:1035] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:46.614885 349 mvcc.cc:205] Tried to move back new op lower bound from 7317106797005766656 to 7317106796104392704. Current Snapshot: MvccSnapshot[applied={T|T < 7317106797005766656}]
I20260810 23:03:46.621843 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.619000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:35519 (local address 127.23.56.66:33227)
0810 23:03:46.619050 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.619052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.619249 (+ 197us) server_negotiation.cc:407] Connection header received
0810 23:03:46.619328 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.619328 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.619341 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.619400 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:46.619561 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.620003 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.621246 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.621426 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.621478 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.621666 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:46.621690 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:46.621726 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.621778 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.621795 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:03:46.621909 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.618961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:33227 (local address 127.23.56.67:35519)
0810 23:03:46.619184 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.619195 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.619254 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.619403 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.619404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.619405 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:46.619476 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.619477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.620092 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.620093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.621152 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.621156 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.621502 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.621503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.621574 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.621588 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:46.621728 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:46.621729 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.621777 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.621816 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":105,"threads_started":1}
I20260810 23:03:46.630146 32141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:46.630255 32141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:46.630925 32141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c: Sending DeleteTablet for 3 replicas of tablet 26cec247893a4ade983b513bda696b11
I20260810 23:03:46.631209 32311 tablet_service.cc:1514] Processing DeleteTablet for tablet 26cec247893a4ade983b513bda696b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:46 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:52904
I20260810 23:03:46.631314 32647 tablet_service.cc:1514] Processing DeleteTablet for tablet 26cec247893a4ade983b513bda696b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:46 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:43878
I20260810 23:03:46.631395 32481 tablet_service.cc:1514] Processing DeleteTablet for tablet 26cec247893a4ade983b513bda696b11 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:46 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:57872
I20260810 23:03:46.631434 370 tablet_replica.cc:330] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: stopping tablet replica
I20260810 23:03:46.631494 370 raft_consensus.cc:2229] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:46.631532 370 raft_consensus.cc:2258] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:46.631702 372 tablet_replica.cc:330] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: stopping tablet replica
I20260810 23:03:46.631799 370 ts_tablet_manager.cc:1905] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:46.631800 372 raft_consensus.cc:2229] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:46.631932 372 raft_consensus.cc:2258] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:46.632217 372 ts_tablet_manager.cc:1905] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:46.633845 370 ts_tablet_manager.cc:1918] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:46.633919 370 log.cc:1196] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/26cec247893a4ade983b513bda696b11
I20260810 23:03:46.634097 372 ts_tablet_manager.cc:1918] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:46.634166 370 ts_tablet_manager.cc:1939] T 26cec247893a4ade983b513bda696b11 P 571a317f77bd45c5bd9b66fb7a478503: Deleting consensus metadata
I20260810 23:03:46.634260 372 log.cc:1196] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/26cec247893a4ade983b513bda696b11
I20260810 23:03:46.634498 32126 catalog_manager.cc:4841] TS 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051): tablet 26cec247893a4ade983b513bda696b11 (table test-table [id=64e5f36b020644ff86028608fa41132f]) successfully deleted
I20260810 23:03:46.634639 372 ts_tablet_manager.cc:1939] T 26cec247893a4ade983b513bda696b11 P 03392a66d2f947c1b966bd13a1efde4f: Deleting consensus metadata
I20260810 23:03:46.634866 373 tablet_replica.cc:330] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: stopping tablet replica
I20260810 23:03:46.634933 373 raft_consensus.cc:2229] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:46.634992 373 raft_consensus.cc:2258] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:46.634971 32127 catalog_manager.cc:4841] TS 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307): tablet 26cec247893a4ade983b513bda696b11 (table test-table [id=64e5f36b020644ff86028608fa41132f]) successfully deleted
I20260810 23:03:46.635257 373 ts_tablet_manager.cc:1905] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:46.637029 373 ts_tablet_manager.cc:1918] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:46.637091 373 log.cc:1196] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/26cec247893a4ade983b513bda696b11
I20260810 23:03:46.637329 373 ts_tablet_manager.cc:1939] T 26cec247893a4ade983b513bda696b11 P b3dbd906239f4875be13f7cfd0e56386: Deleting consensus metadata
I20260810 23:03:46.637645 32125 catalog_manager.cc:4841] TS b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227): tablet 26cec247893a4ade983b513bda696b11 (table test-table [id=64e5f36b020644ff86028608fa41132f]) successfully deleted
Aug 10 23:03:46 dist-test-slave-1xrh krb5kdc[32092](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 23:03:46 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:46 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:03:46.659584 32205 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:46.653486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.23.56.126:46733)
0810 23:03:46.653570 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.653573 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.653696 (+ 123us) server_negotiation.cc:407] Connection header received
0810 23:03:46.653768 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.653769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.653779 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.653826 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:46.654504 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.654977 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.656057 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.656242 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.656293 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.659511 (+ 3218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52454: BlockingRecv error: recv got EOF from 127.0.0.1:52454 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 23:03:46 dist-test-slave-1xrh krb5kdc[32092](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403026, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM
I20260810 23:03:46.667225 32205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.660439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.23.56.126:46733)
0810 23:03:46.660480 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.660482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.660581 (+ 99us) server_negotiation.cc:407] Connection header received
0810 23:03:46.660655 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.660656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.660666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.660732 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:46.661199 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.661666 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.662345 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.662541 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.662612 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.664726 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:46.664727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:46.664728 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:46.664736 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:46.666081 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:46.666420 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:46.666421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:46.666421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:46.666491 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:46.666691 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:46.666692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:46.666692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:46.666801 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:46.666908 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.667003 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.667083 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:03:46.668962 32141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:46.669129 32141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:46.670410 32311 tablet_service.cc:1467] Processing CreateTablet for tablet 0c38bb61ca954293a56bfd876a1607f0 (DEFAULT_TABLE table=test-table [id=ec83ebda451f4c11a79df15fa3990653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.670480 32481 tablet_service.cc:1467] Processing CreateTablet for tablet 0c38bb61ca954293a56bfd876a1607f0 (DEFAULT_TABLE table=test-table [id=ec83ebda451f4c11a79df15fa3990653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.670549 32311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c38bb61ca954293a56bfd876a1607f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.670605 32481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c38bb61ca954293a56bfd876a1607f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.672053 32763 tablet_bootstrap.cc:492] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: Bootstrap starting.
I20260810 23:03:46.672092 32760 tablet_bootstrap.cc:492] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: Bootstrap starting.
I20260810 23:03:46.672260 32647 tablet_service.cc:1467] Processing CreateTablet for tablet 0c38bb61ca954293a56bfd876a1607f0 (DEFAULT_TABLE table=test-table [id=ec83ebda451f4c11a79df15fa3990653]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.672370 32647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c38bb61ca954293a56bfd876a1607f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.672417 32763 tablet_bootstrap.cc:654] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.672459 32760 tablet_bootstrap.cc:654] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.673058 32760 tablet_bootstrap.cc:492] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: No bootstrap required, opened a new log
I20260810 23:03:46.673105 32760 ts_tablet_manager.cc:1397] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:46.673240 32760 raft_consensus.cc:348] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.673301 32760 raft_consensus.cc:374] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.673321 32760 raft_consensus.cc:729] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571a317f77bd45c5bd9b66fb7a478503, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.673369 32760 consensus_queue.cc:260] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.673452 32760 ts_tablet_manager.cc:1428] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:46.673729 32761 tablet_bootstrap.cc:492] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: Bootstrap starting.
I20260810 23:03:46.674113 32761 tablet_bootstrap.cc:654] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.674801 32761 tablet_bootstrap.cc:492] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: No bootstrap required, opened a new log
I20260810 23:03:46.674857 32761 ts_tablet_manager.cc:1397] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:46.675027 32761 raft_consensus.cc:348] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.675112 32761 raft_consensus.cc:374] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.675164 32761 raft_consensus.cc:729] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03392a66d2f947c1b966bd13a1efde4f, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.675212 32761 consensus_queue.cc:260] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.675237 32763 tablet_bootstrap.cc:492] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: No bootstrap required, opened a new log
I20260810 23:03:46.675277 32763 ts_tablet_manager.cc:1397] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 23:03:46.675338 32761 ts_tablet_manager.cc:1428] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260810 23:03:46.675386 32763 raft_consensus.cc:348] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.675441 32763 raft_consensus.cc:374] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.675465 32763 raft_consensus.cc:729] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3dbd906239f4875be13f7cfd0e56386, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.675522 32763 consensus_queue.cc:260] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.675618 32763 ts_tablet_manager.cc:1428] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:46.783617 304 raft_consensus.cc:482] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:46.783727 304 raft_consensus.cc:504] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.784160 304 leader_election.cc:288] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227), 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307)
I20260810 23:03:46.788180 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.784446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:48151 (local address 127.23.56.67:46307)
0810 23:03:46.784532 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.784534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.784682 (+ 148us) server_negotiation.cc:407] Connection header received
0810 23:03:46.784771 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.784772 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.784786 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.784851 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:46.785054 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.785707 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.787252 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.787820 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.787888 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.787897 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.788073 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.788115 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:03:46.788180 340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.784496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:52245 (local address 127.23.56.66:33227)
0810 23:03:46.784547 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.784549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.784700 (+ 151us) server_negotiation.cc:407] Connection header received
0810 23:03:46.784774 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.784775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.784786 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.784844 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:03:46.785344 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.785955 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.787252 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.787795 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.787878 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.787888 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.788066 (+ 178us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.788106 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:03:46.788237 391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.784349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:46307 (local address 127.23.56.65:48151)
0810 23:03:46.784596 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.784608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.784691 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.784868 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.784869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.784873 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:46.784977 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.784979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.785789 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.785789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.787164 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.787168 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.787904 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.787906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.787985 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.787992 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.788054 (+ 62us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.788115 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":62,"thread_start_us":158,"threads_started":1}
I20260810 23:03:46.788439 392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.784357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:33227 (local address 127.23.56.65:52245)
0810 23:03:46.784596 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:46.784608 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:46.784682 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:46.784868 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:46.784869 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:46.784871 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:03:46.785266 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.785268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.786052 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.786053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.787169 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:46.787171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.787906 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.787907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:46.787985 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.788003 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 23:03:46.788054 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 23:03:46.788115 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":53,"thread_start_us":168,"threads_started":1}
I20260810 23:03:46.788508 32674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c38bb61ca954293a56bfd876a1607f0" candidate_uuid: "571a317f77bd45c5bd9b66fb7a478503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03392a66d2f947c1b966bd13a1efde4f" is_pre_election: true
I20260810 23:03:46.788614 32674 raft_consensus.cc:2454] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 571a317f77bd45c5bd9b66fb7a478503 in term 0.
I20260810 23:03:46.788730 32507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c38bb61ca954293a56bfd876a1607f0" candidate_uuid: "571a317f77bd45c5bd9b66fb7a478503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dbd906239f4875be13f7cfd0e56386" is_pre_election: true
I20260810 23:03:46.788817 32507 raft_consensus.cc:2454] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 571a317f77bd45c5bd9b66fb7a478503 in term 0.
I20260810 23:03:46.788792 32252 leader_election.cc:302] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03392a66d2f947c1b966bd13a1efde4f, 571a317f77bd45c5bd9b66fb7a478503; no voters:
I20260810 23:03:46.788931 304 raft_consensus.cc:2781] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:46.789031 304 raft_consensus.cc:482] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:46.789067 304 raft_consensus.cc:3037] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.789867 304 raft_consensus.cc:504] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.790022 304 leader_election.cc:288] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [CANDIDATE]: Term 1 election: Requested vote from peers b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227), 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307)
I20260810 23:03:46.790169 32507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c38bb61ca954293a56bfd876a1607f0" candidate_uuid: "571a317f77bd45c5bd9b66fb7a478503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dbd906239f4875be13f7cfd0e56386"
I20260810 23:03:46.790247 32507 raft_consensus.cc:3037] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.790274 32674 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c38bb61ca954293a56bfd876a1607f0" candidate_uuid: "571a317f77bd45c5bd9b66fb7a478503" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "03392a66d2f947c1b966bd13a1efde4f"
I20260810 23:03:46.790351 32674 raft_consensus.cc:3037] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:46.791044 32674 raft_consensus.cc:2454] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571a317f77bd45c5bd9b66fb7a478503 in term 1.
I20260810 23:03:46.791044 32507 raft_consensus.cc:2454] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 571a317f77bd45c5bd9b66fb7a478503 in term 1.
I20260810 23:03:46.791211 32252 leader_election.cc:302] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03392a66d2f947c1b966bd13a1efde4f, 571a317f77bd45c5bd9b66fb7a478503; no voters:
I20260810 23:03:46.791326 304 raft_consensus.cc:2781] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:46.791455 304 raft_consensus.cc:686] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 LEADER]: Becoming Leader. State: Replica: 571a317f77bd45c5bd9b66fb7a478503, State: Running, Role: LEADER
I20260810 23:03:46.791535 304 consensus_queue.cc:237] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.792246 32143 catalog_manager.cc:5495] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 reported cstate change: term changed from 0 to 1, leader changed from <none> to 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65). New cstate: current_term: 1 leader_uuid: "571a317f77bd45c5bd9b66fb7a478503" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } health_report { overall_health: UNKNOWN } } }
I20260810 23:03:46.819835 339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:46.817268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52910 (local address 127.23.56.65:33051)
0810 23:03:46.817327 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:46.817328 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:46.817337 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:46.817429 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:46.817429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:46.817450 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:46.817517 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:46.817680 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.818122 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.818929 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:46.819118 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:46.819256 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:46.819451 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:46.819568 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:46.819657 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:46.819698 (+ 41us) server_negotiation.cc:299] Negotiation successful
0810 23:03:46.819722 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:03:46.820932 32674 raft_consensus.cc:1261] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Refusing update from remote peer 571a317f77bd45c5bd9b66fb7a478503: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:46.820952 32507 raft_consensus.cc:1261] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Refusing update from remote peer 571a317f77bd45c5bd9b66fb7a478503: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:46.821193 304 consensus_queue.cc:1035] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:46.821349 393 consensus_queue.cc:1035] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:46.826052 32141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:46.826156 32141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:46.826866 32141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c: Sending DeleteTablet for 3 replicas of tablet 0c38bb61ca954293a56bfd876a1607f0
I20260810 23:03:46.827109 32481 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c38bb61ca954293a56bfd876a1607f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:46 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:57872
I20260810 23:03:46.827172 32311 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c38bb61ca954293a56bfd876a1607f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:46 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:52904
I20260810 23:03:46.827224 373 tablet_replica.cc:330] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: stopping tablet replica
I20260810 23:03:46.827247 370 tablet_replica.cc:330] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: stopping tablet replica
I20260810 23:03:46.827270 373 raft_consensus.cc:2229] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:46.827289 370 raft_consensus.cc:2229] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:46.827333 373 raft_consensus.cc:2258] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:46.827360 370 raft_consensus.cc:2258] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:46.827402 32141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:03:46.827561 32141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:03:46.827566 32647 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c38bb61ca954293a56bfd876a1607f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:46 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:43878
I20260810 23:03:46.827777 373 ts_tablet_manager.cc:1905] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:46.828837 32647 tablet_service.cc:1467] Processing CreateTablet for tablet 4e53189f3e4645fdaf45d7424dc97677 (DEFAULT_TABLE table=test-table [id=70caf540f57e4e2ba143b9205c6257e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.828953 32647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e53189f3e4645fdaf45d7424dc97677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.828966 32311 tablet_service.cc:1467] Processing CreateTablet for tablet 4e53189f3e4645fdaf45d7424dc97677 (DEFAULT_TABLE table=test-table [id=70caf540f57e4e2ba143b9205c6257e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.829092 32311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e53189f3e4645fdaf45d7424dc97677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.829414 372 tablet_replica.cc:330] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: stopping tablet replica
I20260810 23:03:46.829468 372 raft_consensus.cc:2229] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:46.829507 372 raft_consensus.cc:2258] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:46.829684 32481 tablet_service.cc:1467] Processing CreateTablet for tablet 4e53189f3e4645fdaf45d7424dc97677 (DEFAULT_TABLE table=test-table [id=70caf540f57e4e2ba143b9205c6257e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:03:46.829833 373 ts_tablet_manager.cc:1918] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:46.829871 373 log.cc:1196] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/0c38bb61ca954293a56bfd876a1607f0
I20260810 23:03:46.830102 373 ts_tablet_manager.cc:1939] T 0c38bb61ca954293a56bfd876a1607f0 P b3dbd906239f4875be13f7cfd0e56386: Deleting consensus metadata
I20260810 23:03:46.830228 32481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e53189f3e4645fdaf45d7424dc97677. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:46.830404 32761 tablet_bootstrap.cc:492] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: Bootstrap starting.
I20260810 23:03:46.830579 32760 tablet_bootstrap.cc:492] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503: Bootstrap starting.
I20260810 23:03:46.830708 32761 tablet_bootstrap.cc:654] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.831028 32760 tablet_bootstrap.cc:654] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.831514 32763 tablet_bootstrap.cc:492] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: Bootstrap starting.
I20260810 23:03:46.831535 32761 tablet_bootstrap.cc:492] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: No bootstrap required, opened a new log
I20260810 23:03:46.831571 32761 ts_tablet_manager.cc:1397] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:46.831679 32761 raft_consensus.cc:348] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.831749 32761 raft_consensus.cc:374] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.831774 32761 raft_consensus.cc:729] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 03392a66d2f947c1b966bd13a1efde4f, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.831811 32763 tablet_bootstrap.cc:654] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:46.831837 32761 consensus_queue.cc:260] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.831943 32761 ts_tablet_manager.cc:1428] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:46.832075 32760 tablet_bootstrap.cc:492] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503: No bootstrap required, opened a new log
I20260810 23:03:46.832131 32760 ts_tablet_manager.cc:1397] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 23:03:46.832264 370 ts_tablet_manager.cc:1905] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:46.832274 32760 raft_consensus.cc:348] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.833431 32760 raft_consensus.cc:374] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.833452 32760 raft_consensus.cc:729] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571a317f77bd45c5bd9b66fb7a478503, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.833495 32760 consensus_queue.cc:260] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.833585 32760 ts_tablet_manager.cc:1428] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:46.833766 370 ts_tablet_manager.cc:1918] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:46.833846 370 log.cc:1196] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/0c38bb61ca954293a56bfd876a1607f0
I20260810 23:03:46.833948 32125 catalog_manager.cc:4841] TS b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227): tablet 0c38bb61ca954293a56bfd876a1607f0 (table test-table [id=ec83ebda451f4c11a79df15fa3990653]) successfully deleted
I20260810 23:03:46.834097 370 ts_tablet_manager.cc:1939] T 0c38bb61ca954293a56bfd876a1607f0 P 571a317f77bd45c5bd9b66fb7a478503: Deleting consensus metadata
I20260810 23:03:46.834192 372 ts_tablet_manager.cc:1905] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:46.834450 32763 tablet_bootstrap.cc:492] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: No bootstrap required, opened a new log
I20260810 23:03:46.834487 32763 ts_tablet_manager.cc:1397] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260810 23:03:46.834578 32126 catalog_manager.cc:4841] TS 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051): tablet 0c38bb61ca954293a56bfd876a1607f0 (table test-table [id=ec83ebda451f4c11a79df15fa3990653]) successfully deleted
I20260810 23:03:46.834576 32763 raft_consensus.cc:348] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.834631 32763 raft_consensus.cc:374] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:46.834651 32763 raft_consensus.cc:729] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3dbd906239f4875be13f7cfd0e56386, State: Initialized, Role: FOLLOWER
I20260810 23:03:46.834694 32763 consensus_queue.cc:260] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:46.834780 32763 ts_tablet_manager.cc:1428] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:46.835568 372 ts_tablet_manager.cc:1918] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:03:46.835618 372 log.cc:1196] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/0c38bb61ca954293a56bfd876a1607f0
I20260810 23:03:46.835827 372 ts_tablet_manager.cc:1939] T 0c38bb61ca954293a56bfd876a1607f0 P 03392a66d2f947c1b966bd13a1efde4f: Deleting consensus metadata
I20260810 23:03:46.836215 32127 catalog_manager.cc:4841] TS 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307): tablet 0c38bb61ca954293a56bfd876a1607f0 (table test-table [id=ec83ebda451f4c11a79df15fa3990653]) successfully deleted
I20260810 23:03:47.047825 365 raft_consensus.cc:482] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:03:47.047928 365 raft_consensus.cc:504] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:47.048130 365 leader_election.cc:288] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051), b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227)
I20260810 23:03:47.048378 32333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e53189f3e4645fdaf45d7424dc97677" candidate_uuid: "03392a66d2f947c1b966bd13a1efde4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571a317f77bd45c5bd9b66fb7a478503" is_pre_election: true
I20260810 23:03:47.048370 32507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e53189f3e4645fdaf45d7424dc97677" candidate_uuid: "03392a66d2f947c1b966bd13a1efde4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dbd906239f4875be13f7cfd0e56386" is_pre_election: true
I20260810 23:03:47.048462 32507 raft_consensus.cc:2454] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03392a66d2f947c1b966bd13a1efde4f in term 0.
I20260810 23:03:47.048481 32333 raft_consensus.cc:2454] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 03392a66d2f947c1b966bd13a1efde4f in term 0.
I20260810 23:03:47.048666 32593 leader_election.cc:302] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03392a66d2f947c1b966bd13a1efde4f, 571a317f77bd45c5bd9b66fb7a478503; no voters:
I20260810 23:03:47.048786 365 raft_consensus.cc:2781] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:03:47.048838 365 raft_consensus.cc:482] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:03:47.048869 365 raft_consensus.cc:3037] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:47.049598 365 raft_consensus.cc:504] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:47.049741 365 leader_election.cc:288] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [CANDIDATE]: Term 1 election: Requested vote from peers 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051), b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227)
I20260810 23:03:47.049914 32333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e53189f3e4645fdaf45d7424dc97677" candidate_uuid: "03392a66d2f947c1b966bd13a1efde4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "571a317f77bd45c5bd9b66fb7a478503"
I20260810 23:03:47.049914 32507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e53189f3e4645fdaf45d7424dc97677" candidate_uuid: "03392a66d2f947c1b966bd13a1efde4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3dbd906239f4875be13f7cfd0e56386"
I20260810 23:03:47.049996 32507 raft_consensus.cc:3037] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:47.049995 32333 raft_consensus.cc:3037] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:47.050782 32507 raft_consensus.cc:2454] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03392a66d2f947c1b966bd13a1efde4f in term 1.
I20260810 23:03:47.050782 32333 raft_consensus.cc:2454] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 03392a66d2f947c1b966bd13a1efde4f in term 1.
I20260810 23:03:47.051002 32593 leader_election.cc:302] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 03392a66d2f947c1b966bd13a1efde4f, 571a317f77bd45c5bd9b66fb7a478503; no voters:
I20260810 23:03:47.051112 365 raft_consensus.cc:2781] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:47.051223 365 raft_consensus.cc:686] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 1 LEADER]: Becoming Leader. State: Replica: 03392a66d2f947c1b966bd13a1efde4f, State: Running, Role: LEADER
I20260810 23:03:47.051278 365 consensus_queue.cc:237] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } }
I20260810 23:03:47.051926 32143 catalog_manager.cc:5495] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67). New cstate: current_term: 1 leader_uuid: "03392a66d2f947c1b966bd13a1efde4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "03392a66d2f947c1b966bd13a1efde4f" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 46307 } health_report { overall_health: HEALTHY } } }
I20260810 23:03:47.065006 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:47.062654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43910 (local address 127.23.56.67:46307)
0810 23:03:47.062695 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:47.062696 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:47.062718 (+ 22us) server_negotiation.cc:407] Connection header received
0810 23:03:47.062784 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:47.062784 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:47.062794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:47.062847 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:47.062965 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:47.063543 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:47.064290 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:47.064506 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:47.064583 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:47.064734 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:47.064767 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:47.064838 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:47.064900 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 23:03:47.064923 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:03:47.068920 348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:47.066944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:44779 (local address 127.23.56.67:46307)
0810 23:03:47.067000 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:47.067002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:47.067010 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:03:47.067031 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:47.067032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:47.067045 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:47.067087 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:03:47.067239 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:47.067649 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:47.068298 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:47.068468 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:47.068516 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:47.068641 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:03:47.068671 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:03:47.068727 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:47.068817 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 23:03:47.068852 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":37}
I20260810 23:03:47.068926 367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:47.066859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:46307 (local address 127.23.56.67:44779)
0810 23:03:47.066911 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:47.066922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:47.066972 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:47.067100 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:47.067101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:47.067102 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 23:03:47.067185 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:47.067187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:47.067705 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:47.067706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:47.068228 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:47.068229 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:47.068533 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:47.068534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:47.068572 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:47.068578 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 23:03:47.068728 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 23:03:47.068729 (+ 1us) client_negotiation.cc:769] Sending connection context
0810 23:03:47.068813 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:03:47.068867 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:03:47.070895 32507 raft_consensus.cc:1261] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Refusing update from remote peer 03392a66d2f947c1b966bd13a1efde4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:47.070936 32333 raft_consensus.cc:1261] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Refusing update from remote peer 03392a66d2f947c1b966bd13a1efde4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:03:47.071105 302 consensus_queue.cc:1035] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3dbd906239f4875be13f7cfd0e56386" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:47.071233 365 consensus_queue.cc:1035] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "571a317f77bd45c5bd9b66fb7a478503" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 33051 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:03:47.081336 32143 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:47.081435 32143 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:03:47.082093 32143 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cceb5ac5bba040c79df96c5dde00f84c: Sending DeleteTablet for 3 replicas of tablet 4e53189f3e4645fdaf45d7424dc97677
I20260810 23:03:47.082377 32646 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e53189f3e4645fdaf45d7424dc97677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:47 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:43878
I20260810 23:03:47.082399 32311 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e53189f3e4645fdaf45d7424dc97677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:47 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:52904
I20260810 23:03:47.082505 32481 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e53189f3e4645fdaf45d7424dc97677 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:03:47 UTC) from {username='oryx', principal='oryx/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:57872
I20260810 23:03:47.082595 372 tablet_replica.cc:330] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: stopping tablet replica
I20260810 23:03:47.082657 372 raft_consensus.cc:2229] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:03:47.082681 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 32229
I20260810 23:03:47.082746 372 raft_consensus.cc:2258] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:47.082829 370 tablet_replica.cc:330] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503: stopping tablet replica
I20260810 23:03:47.082885 370 raft_consensus.cc:2229] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:47.082916 370 raft_consensus.cc:2258] T 4e53189f3e4645fdaf45d7424dc97677 P 571a317f77bd45c5bd9b66fb7a478503 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:47.082980 373 tablet_replica.cc:330] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: stopping tablet replica
I20260810 23:03:47.083030 373 raft_consensus.cc:2229] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:03:47.083061 373 raft_consensus.cc:2258] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:03:47.085215 372 ts_tablet_manager.cc:1905] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:47.085557 373 ts_tablet_manager.cc:1905] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:03:47.087240 373 ts_tablet_manager.cc:1918] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:47.087307 373 log.cc:1196] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/4e53189f3e4645fdaf45d7424dc97677
I20260810 23:03:47.087607 373 ts_tablet_manager.cc:1939] T 4e53189f3e4645fdaf45d7424dc97677 P b3dbd906239f4875be13f7cfd0e56386: Deleting consensus metadata
I20260810 23:03:47.087927 32125 catalog_manager.cc:4841] TS b3dbd906239f4875be13f7cfd0e56386 (127.23.56.66:33227): tablet 4e53189f3e4645fdaf45d7424dc97677 (table test-table [id=70caf540f57e4e2ba143b9205c6257e2]) successfully deleted
I20260810 23:03:47.088199 372 ts_tablet_manager.cc:1918] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 23:03:47.088258 372 log.cc:1196] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/4e53189f3e4645fdaf45d7424dc97677
I20260810 23:03:47.088523 372 ts_tablet_manager.cc:1939] T 4e53189f3e4645fdaf45d7424dc97677 P 03392a66d2f947c1b966bd13a1efde4f: Deleting consensus metadata
I20260810 23:03:47.088907 32127 catalog_manager.cc:4841] TS 03392a66d2f947c1b966bd13a1efde4f (127.23.56.67:46307): tablet 4e53189f3e4645fdaf45d7424dc97677 (table test-table [id=70caf540f57e4e2ba143b9205c6257e2]) successfully deleted
W20260810 23:03:47.094120 32126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:33051 (error 108)
I20260810 23:03:47.094695 410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:47.094511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:33051 (local address 127.0.0.1:52916)
0810 23:03:47.094602 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:47.094647 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:33051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":47,"threads_started":1}
I20260810 23:03:47.094831 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 32401
W20260810 23:03:47.094944 32126 catalog_manager.cc:4584] TS 571a317f77bd45c5bd9b66fb7a478503 (127.23.56.65:33051): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e53189f3e4645fdaf45d7424dc97677: Network error: Client connection negotiation failed: client connection to 127.23.56.65:33051: connect: Connection refused (error 111)
I20260810 23:03:47.103907 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 32573
I20260810 23:03:47.113724 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 32106
2026-08-10T23:03:47Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2998 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:47 dist-test-slave-1xrh krb5kdc[413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:47 dist-test-slave-1xrh krb5kdc[413](info): set up 2 sockets
Aug 10 23:03:47 dist-test-slave-1xrh krb5kdc[413](info): commencing operation
krb5kdc: starting...
W20260810 23:03:48.286898 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.149s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:48 dist-test-slave-1xrh krb5kdc[413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:48.418854 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34959
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:34959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:48.500612 428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:48.500844 428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:48.500880 428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:48.502043 428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:48.502106 428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:48.502131 428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:48.502170 428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:48.502195 428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:48.503713 428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:34959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:48.503995 428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:48.504225 428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:48.509091 435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:48.509102 434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:48.509161 437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:48.509419 428 server_base.cc:1034] running on GCE node
I20260810 23:03:48.509666 428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:48.509902 428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:48.511044 428 hybrid_clock.cc:648] HybridClock initialized: now 1786403028511029 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T23:03:48Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1399 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:03:48 dist-test-slave-1xrh krb5kdc[443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:03:48 dist-test-slave-1xrh krb5kdc[443](info): set up 2 sockets
Aug 10 23:03:48 dist-test-slave-1xrh krb5kdc[443](info): commencing operation
krb5kdc: starting...
W20260810 23:03:49.608826 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:03:49 dist-test-slave-1xrh krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:03:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:03:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:03:49.731035 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33467
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:49.812628 458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:49.812865 458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:49.812902 458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:49.814043 458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:49.814112 458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:49.814139 458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:49.814170 458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:49.814193 458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:49.815683 458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:49.815961 458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:49.816196 458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:49.821089 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:49.821071 466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:49.821101 458 server_base.cc:1034] running on GCE node
W20260810 23:03:49.821059 468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:49.821488 458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:49.821767 458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:49.822935 458 hybrid_clock.cc:648] HybridClock initialized: now 1786403029822915 us; error 36 us; skew 500 ppm
Aug 10 23:03:49 dist-test-slave-1xrh krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403029, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:49.825439 458 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:49.825898 458 webserver.cc:460] Webserver started at http://127.23.56.126:41487/ using document root <none> and password file <none>
I20260810 23:03:49.826121 458 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:49.826188 458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:49.826344 458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:49.827462 458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "0350ecb533f54902ab623d5575a6e0ce"
format_stamp: "Formatted at 2026-08-10 23:03:49 on dist-test-slave-1xrh"
server_key: "3072776d070e3e0e48f498f4cfcfbfec"
server_key_iv: "c348381efd09fd1f77d98702bc46ce37"
server_key_version: "encryptionkey@0"
I20260810 23:03:49.827826 458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "0350ecb533f54902ab623d5575a6e0ce"
format_stamp: "Formatted at 2026-08-10 23:03:49 on dist-test-slave-1xrh"
server_key: "3072776d070e3e0e48f498f4cfcfbfec"
server_key_iv: "c348381efd09fd1f77d98702bc46ce37"
server_key_version: "encryptionkey@0"
I20260810 23:03:49.829129 458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:49.829829 477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:49.829998 458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:49.830094 458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "0350ecb533f54902ab623d5575a6e0ce"
format_stamp: "Formatted at 2026-08-10 23:03:49 on dist-test-slave-1xrh"
server_key: "3072776d070e3e0e48f498f4cfcfbfec"
server_key_iv: "c348381efd09fd1f77d98702bc46ce37"
server_key_version: "encryptionkey@0"
I20260810 23:03:49.830176 458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:49.837579 458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:49.839501 458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:49.839608 458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:49.843482 458 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33467
I20260810 23:03:49.843534 538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33467 every 8 connection(s)
I20260810 23:03:49.843889 458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:49.844724 539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:03:49.845821 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 458
I20260810 23:03:49.845942 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:03:49.846143 23777 external_mini_cluster.cc:1428] Setting key 1a585d472d24142462deb2dee5e595c6
I20260810 23:03:49.847570 539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: Bootstrap starting.
I20260810 23:03:49.848300 539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: Neither blocks nor log segments found. Creating new log.
I20260810 23:03:49.848625 539 log.cc:824] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:49.849300 539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: No bootstrap required, opened a new log
I20260810 23:03:49.851033 539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } }
I20260810 23:03:49.851236 539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:03:49.851290 539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0350ecb533f54902ab623d5575a6e0ce, State: Initialized, Role: FOLLOWER
I20260810 23:03:49.851438 539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } }
I20260810 23:03:49.851526 539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:49.851572 539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:49.851619 539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:03:49.852133 539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } }
I20260810 23:03:49.852248 539 leader_election.cc:302] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0350ecb533f54902ab623d5575a6e0ce; no voters:
I20260810 23:03:49.852452 539 leader_election.cc:288] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:03:49.852475 544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:03:49.852602 544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 1 LEADER]: Becoming Leader. State: Replica: 0350ecb533f54902ab623d5575a6e0ce, State: Running, Role: LEADER
I20260810 23:03:49.852694 544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } }
I20260810 23:03:49.852725 539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 23:03:49 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:49.853045 544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0350ecb533f54902ab623d5575a6e0ce. Latest consensus state: current_term: 1 leader_uuid: "0350ecb533f54902ab623d5575a6e0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } } }
I20260810 23:03:49.853129 544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:49.853260 549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0350ecb533f54902ab623d5575a6e0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } } }
I20260810 23:03:49.853346 549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: This master's current role is: LEADER
W20260810 23:03:49.854791 564 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:03:49.854866 564 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:03:49.854933 553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:49.855098 553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:49.856709 553 catalog_manager.cc:1347] Generated new cluster ID: 52047360d7434d59806697fa85b0e7dd
I20260810 23:03:49.856755 553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:49.857060 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:49.846668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48770 (local address 127.23.56.126:33467)
0810 23:03:49.846835 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:49.846844 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:49.846878 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:03:49.847354 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:49.847360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:49.847408 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:49.847533 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:49.848334 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:49.849298 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:49.850070 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:49.850486 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:49.850604 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:49.853315 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:49.853319 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:49.853320 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:49.853331 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:49.855429 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:49.855800 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:49.855804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:49.855804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:49.855904 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:49.856143 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:49.856144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:49.856144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:49.856438 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:49.856609 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:49.856719 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 23:03:49.856852 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260810 23:03:49.869419 553 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:03:49.870013 553 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:49.881275 553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: Generated new TSK 0
I20260810 23:03:49.881475 553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:03:49.914270 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33467
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:50.004487 569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:50.004717 569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:50.004743 569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:50.005945 569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:50.005987 569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:50.006016 569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:50.007542 569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:50.007807 569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:50.008039 569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:50.013146 579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.013234 569 server_base.cc:1034] running on GCE node
W20260810 23:03:50.013164 577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.013195 574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.013671 569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:50.013933 569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:50.015071 569 hybrid_clock.cc:648] HybridClock initialized: now 1786403030015053 us; error 38 us; skew 500 ppm
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:50.017670 569 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:03:50.018090 569 webserver.cc:460] Webserver started at http://127.23.56.65:34937/ using document root <none> and password file <none>
I20260810 23:03:50.018319 569 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:50.018365 569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:50.018549 569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:50.019656 569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "991a8f87b3174f10bac63ccd44154283"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "c2724db454da9a69e8e1edba5749da4b"
server_key_iv: "875788a992b6940e1ecc8aefce01ea5f"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.020020 569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "991a8f87b3174f10bac63ccd44154283"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "c2724db454da9a69e8e1edba5749da4b"
server_key_iv: "875788a992b6940e1ecc8aefce01ea5f"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.021391 569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:50.022234 589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.022416 569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:50.022517 569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "991a8f87b3174f10bac63ccd44154283"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "c2724db454da9a69e8e1edba5749da4b"
server_key_iv: "875788a992b6940e1ecc8aefce01ea5f"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.022603 569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:50.039270 569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:50.041141 569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:50.041304 569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:50.041533 569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:50.041849 569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:50.041903 569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.041944 569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:50.041990 569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.047895 569 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:45309
I20260810 23:03:50.047948 723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:45309 every 8 connection(s)
I20260810 23:03:50.048267 569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:03:50.050235 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 569
I20260810 23:03:50.050323 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:03:50.050496 23777 external_mini_cluster.cc:1428] Setting key e858679e7ef0b043c2cbc7907d63f061
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:50.058561 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.048976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:45019 (local address 127.23.56.126:33467)
0810 23:03:50.049071 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.049074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.049223 (+ 149us) server_negotiation.cc:407] Connection header received
0810 23:03:50.049700 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.049701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.049712 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.049762 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.050825 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.051394 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.052403 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.052649 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.052714 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.055392 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:50.055394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:50.055394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:50.055403 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:50.057155 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:50.057552 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:50.057553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:50.057553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:50.057619 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:50.057856 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:50.057857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:50.057857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:50.058004 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:50.058077 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:50.058332 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 23:03:50.058438 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:03:50.058583 726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.048929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:45019)
0810 23:03:50.049114 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:50.049150 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:50.049615 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:50.049815 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:50.049818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:50.050473 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:50.050726 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.050732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.051489 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.051491 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.052307 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.052309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.052794 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.052796 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.052952 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.053621 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:50.053634 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:50.055291 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:50.057244 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:50.057251 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:50.057253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:50.057475 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:50.057689 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:50.057694 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:50.057694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:50.057791 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:50.058075 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:50.058081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:50.058196 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 23:03:50.058284 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:03:50.058421 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":100,"threads_started":1}
I20260810 23:03:50.059216 724 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33467
I20260810 23:03:50.059322 724 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:50.059536 724 heartbeater.cc:511] Master 127.23.56.126:33467 requested a full tablet report, sending...
I20260810 23:03:50.060086 497 ts_manager.cc:194] Registered new tserver with Master: 991a8f87b3174f10bac63ccd44154283 (127.23.56.65:45309)
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
I20260810 23:03:50.060747 497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:45019
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:03:50.079005 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33467
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:50.161326 729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:50.161558 729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:50.161597 729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:50.162750 729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:50.162810 729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:50.162858 729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:50.164364 729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:50.164633 729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:50.164860 729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:50.169837 735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.170053 729 server_base.cc:1034] running on GCE node
W20260810 23:03:50.169816 736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.169816 738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.170400 729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:50.170608 729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:50.171770 729 hybrid_clock.cc:648] HybridClock initialized: now 1786403030171731 us; error 51 us; skew 500 ppm
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:50.174325 729 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:03:50.174701 729 webserver.cc:460] Webserver started at http://127.23.56.66:38367/ using document root <none> and password file <none>
I20260810 23:03:50.174876 729 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:50.174917 729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:50.175001 729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:50.176083 729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "200c805a3b0049dfb0c416c1610f6631"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "3aa0ee3606c2b642e16ef5bfa3bc9f37"
server_key_iv: "9097bae55789bc1ef72a6c57c2ca9b3f"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.176397 729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "200c805a3b0049dfb0c416c1610f6631"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "3aa0ee3606c2b642e16ef5bfa3bc9f37"
server_key_iv: "9097bae55789bc1ef72a6c57c2ca9b3f"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.177685 729 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:03:50.178339 746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.178498 729 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:50.178577 729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "200c805a3b0049dfb0c416c1610f6631"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "3aa0ee3606c2b642e16ef5bfa3bc9f37"
server_key_iv: "9097bae55789bc1ef72a6c57c2ca9b3f"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.178629 729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:50.193985 729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:50.195820 729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:50.195940 729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:50.196115 729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:50.196365 729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:50.196396 729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.196419 729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:50.196431 729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.202078 729 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:34149
I20260810 23:03:50.202158 890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:34149 every 8 connection(s)
I20260810 23:03:50.202478 729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:50.204075 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 729
I20260810 23:03:50.204192 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:03:50.204367 23777 external_mini_cluster.cc:1428] Setting key 108ac41c2ce89c68cb44df958996b51d
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:50.213092 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.203326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:55785 (local address 127.23.56.126:33467)
0810 23:03:50.203383 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.203385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.203635 (+ 250us) server_negotiation.cc:407] Connection header received
0810 23:03:50.204146 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.204147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.204159 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.204212 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.205465 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.205966 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.207244 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.207446 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.207507 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.210164 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:50.210166 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:50.210166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:50.210173 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:50.211642 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:50.212134 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:50.212135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:50.212135 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:50.212186 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:50.212471 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:50.212472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:50.212472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:50.212574 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:50.212621 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:50.212958 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 23:03:50.213025 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 23:03:50.213382 893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.203292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:55785)
0810 23:03:50.203506 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:50.203555 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:50.204039 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:50.204281 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:50.204290 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:50.205108 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:50.205370 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.205376 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.206073 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.206075 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.207163 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.207166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.207536 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.207537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.207695 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.208251 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:50.208261 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:50.210069 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:50.211725 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:50.211733 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:50.211735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:50.212053 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:50.212255 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:50.212255 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:50.212256 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:50.212399 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:50.212639 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:50.212646 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:50.212814 (+ 168us) client_negotiation.cc:769] Sending connection context
0810 23:03:50.212941 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 23:03:50.213115 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":99,"threads_started":1}
I20260810 23:03:50.213919 891 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33467
I20260810 23:03:50.214022 891 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
I20260810 23:03:50.214236 891 heartbeater.cc:511] Master 127.23.56.126:33467 requested a full tablet report, sending...
I20260810 23:03:50.214622 497 ts_manager.cc:194] Registered new tserver with Master: 200c805a3b0049dfb0c416c1610f6631 (127.23.56.66:34149)
I20260810 23:03:50.214998 497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:55785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:03:50.234309 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33467
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:50.316665 899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:50.316870 899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:50.316895 899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:50.317991 899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:50.318024 899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:50.318050 899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:50.319414 899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:50.319617 899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:50.319800 899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:50.324611 908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.324611 910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.324666 906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.324860 899 server_base.cc:1034] running on GCE node
I20260810 23:03:50.325047 899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:50.325235 899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:50.326381 899 hybrid_clock.cc:648] HybridClock initialized: now 1786403030326368 us; error 28 us; skew 500 ppm
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:50.328989 899 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:03:50.329443 899 webserver.cc:460] Webserver started at http://127.23.56.67:37807/ using document root <none> and password file <none>
I20260810 23:03:50.329648 899 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:50.329687 899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:50.329833 899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:03:50.330816 899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "5a60bef2176749909241627ea4f98acc"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "e8153dc8626bfe2e9bb00d39a99a9e77"
server_key_iv: "25261bd7df583be91117be52a111e6ae"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.331108 899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "5a60bef2176749909241627ea4f98acc"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "e8153dc8626bfe2e9bb00d39a99a9e77"
server_key_iv: "25261bd7df583be91117be52a111e6ae"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.332418 899 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:50.333166 921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.333325 899 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:50.333433 899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "5a60bef2176749909241627ea4f98acc"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "e8153dc8626bfe2e9bb00d39a99a9e77"
server_key_iv: "25261bd7df583be91117be52a111e6ae"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.333516 899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:50.352659 899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:50.354846 899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:50.355005 899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:50.355252 899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:50.355561 899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:50.355612 899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.355654 899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:50.355680 899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.361505 899 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:44273
I20260810 23:03:50.361548 1062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:44273 every 8 connection(s)
I20260810 23:03:50.361855 899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:03:50.369937 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 899
I20260810 23:03:50.370038 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:03:50.370252 23777 external_mini_cluster.cc:1428] Setting key c23f17e24841d404b19a271383b0b45d
I20260810 23:03:50.370791 542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.362482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:49875 (local address 127.23.56.126:33467)
0810 23:03:50.362550 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.362552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.362756 (+ 204us) server_negotiation.cc:407] Connection header received
0810 23:03:50.363357 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.363359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.363371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.363434 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.364375 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.364896 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.365885 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.366086 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.366160 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.368173 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:50.368175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:50.368175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:50.368183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:50.369277 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:50.369664 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:50.369665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:50.369666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:50.369723 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:50.369978 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:50.369979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:50.369979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:50.370167 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:50.370250 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:50.370600 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 23:03:50.370688 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:03:50.370949 1066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.362472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:49875)
0810 23:03:50.362632 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:50.362674 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:50.363257 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:50.363500 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:50.363503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:50.364084 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:50.364264 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.364268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.365005 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.365006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.365785 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.365786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.366167 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.366168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.366306 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.366719 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:50.366727 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:50.368081 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:03:50.369369 (+ 1288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:50.369376 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:50.369378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:50.369588 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:50.369802 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:03:50.369803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:03:50.369803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:03:50.369903 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:03:50.370243 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:03:50.370249 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:03:50.370406 (+ 157us) client_negotiation.cc:769] Sending connection context
0810 23:03:50.370537 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 23:03:50.370709 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260810 23:03:50.371487 1064 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33467
I20260810 23:03:50.371598 1064 heartbeater.cc:464] Registering TS with master...
I20260810 23:03:50.371773 1064 heartbeater.cc:511] Master 127.23.56.126:33467 requested a full tablet report, sending...
I20260810 23:03:50.372134 497 ts_manager.cc:194] Registered new tserver with Master: 5a60bef2176749909241627ea4f98acc (127.23.56.67:44273)
I20260810 23:03:50.372500 497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:49875
I20260810 23:03:50.382102 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab.
Entry for principal oryx/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab.
WARNING: no policy specified for oryx/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab.
Entry for principal oryx/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab.
WARNING: no policy specified for oryx/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab.
Entry for principal oryx/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab.
I20260810 23:03:50.455174 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 569
I20260810 23:03:50.462656 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 729
I20260810 23:03:50.468259 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 899
I20260810 23:03:50.475275 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 458
I20260810 23:03:50.482102 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33467
--webserver_interface=127.23.56.126
--webserver_port=41487
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:03:50.564002 1076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:50.564195 1076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:50.564216 1076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:50.565349 1076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:03:50.565388 1076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:50.565402 1076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:50.565414 1076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:03:50.565426 1076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:03:50.566862 1076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:50.567106 1076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:50.567359 1076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:50.572057 1083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.572177 1081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.572057 1087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.572278 1076 server_base.cc:1034] running on GCE node
I20260810 23:03:50.572569 1076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:50.572871 1076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:50.574046 1076 hybrid_clock.cc:648] HybridClock initialized: now 1786403030574011 us; error 41 us; skew 500 ppm
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:50.580550 1076 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:03:50.580981 1076 webserver.cc:460] Webserver started at http://127.23.56.126:41487/ using document root <none> and password file <none>
I20260810 23:03:50.581200 1076 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:50.581277 1076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:50.582592 1076 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:50.583247 1094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.583415 1076 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:50.583529 1076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "0350ecb533f54902ab623d5575a6e0ce"
format_stamp: "Formatted at 2026-08-10 23:03:49 on dist-test-slave-1xrh"
server_key: "3072776d070e3e0e48f498f4cfcfbfec"
server_key_iv: "c348381efd09fd1f77d98702bc46ce37"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.583869 1076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:50.622146 1076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:50.624471 1076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:50.624655 1076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:50.628466 1076 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33467
I20260810 23:03:50.628511 1156 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33467 every 8 connection(s)
I20260810 23:03:50.628901 1076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:03:50.630407 1157 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:03:50.631063 1157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: Bootstrap starting.
I20260810 23:03:50.637485 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 1076
I20260810 23:03:50.637835 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:45309
--local_ip_for_outbound_sockets=127.23.56.65
--tserver_master_addrs=127.23.56.126:33467
--webserver_port=34937
--webserver_interface=127.23.56.65
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.56.65
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:03:50.640046 1157 log.cc:824] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: Log is configured to *not* fsync() on all Append() calls
I20260810 23:03:50.641776 1157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:03:50.642132 1157 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: Bootstrap complete.
I20260810 23:03:50.644474 1157 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } }
I20260810 23:03:50.644744 1157 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0350ecb533f54902ab623d5575a6e0ce, State: Initialized, Role: FOLLOWER
I20260810 23:03:50.644948 1157 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } }
I20260810 23:03:50.645047 1157 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:03:50.645102 1157 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:03:50.645166 1157 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:03:50.646159 1157 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } }
I20260810 23:03:50.646277 1157 leader_election.cc:302] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0350ecb533f54902ab623d5575a6e0ce; no voters:
I20260810 23:03:50.646512 1157 leader_election.cc:288] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:03:50.646543 1161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:03:50.646664 1161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [term 2 LEADER]: Becoming Leader. State: Replica: 0350ecb533f54902ab623d5575a6e0ce, State: Running, Role: LEADER
I20260810 23:03:50.646811 1157 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:03:50.646829 1161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } }
I20260810 23:03:50.647249 1163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0350ecb533f54902ab623d5575a6e0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } } }
I20260810 23:03:50.647337 1163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:50.647619 1162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0350ecb533f54902ab623d5575a6e0ce. Latest consensus state: current_term: 2 leader_uuid: "0350ecb533f54902ab623d5575a6e0ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0350ecb533f54902ab623d5575a6e0ce" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33467 } } }
I20260810 23:03:50.647751 1162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce [sys.catalog]: This master's current role is: LEADER
I20260810 23:03:50.647720 1168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:03:50.648387 1168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:03:50.648690 1168 catalog_manager.cc:1259] Loaded cluster ID: 52047360d7434d59806697fa85b0e7dd
I20260810 23:03:50.648769 1168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:03:50.650122 1168 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:03:50.650288 1168 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0350ecb533f54902ab623d5575a6e0ce: Loaded TSK: 0
I20260810 23:03:50.650498 1168 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:03:50.722368 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:50.722568 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:50.722594 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:50.723884 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:50.723928 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:50.723955 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:03:50.725399 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.65.keytab
--principal=oryx/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:45309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=34937
--tserver_master_addrs=127.23.56.126:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:50.725646 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:50.725901 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:50.730823 1189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.730926 1193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.730824 1190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.731220 1159 server_base.cc:1034] running on GCE node
I20260810 23:03:50.731496 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:50.731756 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:50.732935 1159 hybrid_clock.cc:648] HybridClock initialized: now 1786403030732914 us; error 39 us; skew 500 ppm
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:50.735980 1159 init.cc:377] Logged in from keytab as oryx/127.23.56.65@KRBTEST.COM (short username oryx)
I20260810 23:03:50.736508 1159 webserver.cc:460] Webserver started at http://127.23.56.65:34937/ using document root <none> and password file <none>
I20260810 23:03:50.736768 1159 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:50.736852 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:50.738238 1159 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:03:50.738852 1201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.739028 1159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 23:03:50.739167 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "991a8f87b3174f10bac63ccd44154283"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "c2724db454da9a69e8e1edba5749da4b"
server_key_iv: "875788a992b6940e1ecc8aefce01ea5f"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.739504 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:50.754462 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:50.756691 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:50.756832 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:50.757094 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:50.757468 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:50.757531 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.757575 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:50.757603 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.765038 1159 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:45309
I20260810 23:03:50.765079 1328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:45309 every 8 connection(s)
I20260810 23:03:50.765444 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:50.774001 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 1159
I20260810 23:03:50.774320 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:34149
--local_ip_for_outbound_sockets=127.23.56.66
--tserver_master_addrs=127.23.56.126:33467
--webserver_port=38367
--webserver_interface=127.23.56.66
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.56.66
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:03:50.774837 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.766238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:36737)
0810 23:03:50.766451 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:50.766505 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:50.766957 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:50.767700 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:50.767704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:50.768273 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:50.768500 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.768506 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.770063 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.770064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.770837 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.770839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.771629 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.771630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.771735 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.772259 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:50.772267 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:50.774612 (+ 2345us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":111,"threads_started":1}
W20260810 23:03:50.774875 1333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:50.766293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:36737 (local address 127.23.56.126:33467)
0810 23:03:50.767267 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.767276 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.767313 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:03:50.767464 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.767471 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.767519 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.767651 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.768627 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.769974 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.770930 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.771532 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.771632 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.774737 (+ 3105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:36737: BlockingRecv error: recv got EOF from 127.23.56.65:36737 (error 108)
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":95,"threads_started":1}
W20260810 23:03:50.775094 1330 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:33467 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.56.126:33467: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:50.779630 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.775382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:35905)
0810 23:03:50.775425 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:50.775436 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:50.775506 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:50.775643 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:50.775644 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:50.775777 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:50.775825 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.775826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.776481 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.776483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.776962 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.776964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.777304 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.777305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.777376 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.777833 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:50.777838 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:50.779548 (+ 1710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 23:03:50.779683 1333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:50.775441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:35905 (local address 127.23.56.126:33467)
0810 23:03:50.775502 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.775504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.775514 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:03:50.775569 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.775570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.775582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.775628 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.775913 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.776401 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.777046 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.777232 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.777305 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.779619 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:35905: BlockingRecv error: recv got EOF from 127.23.56.65:35905 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:50.784793 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.779890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:57733)
0810 23:03:50.780130 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:50.780144 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:50.780206 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:50.780348 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:50.780349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:50.780504 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:50.780542 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.780543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.781141 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.781143 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.782009 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.782012 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.782354 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.782356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.782422 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.782818 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:50.782823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:50.784709 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":220}
W20260810 23:03:50.784904 1333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:50.779971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:57733 (local address 127.23.56.126:33467)
0810 23:03:50.780002 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.780004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.780210 (+ 206us) server_negotiation.cc:407] Connection header received
0810 23:03:50.780268 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.780268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.780277 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.780330 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.780619 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.781063 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.782102 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.782275 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.782338 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.784782 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:57733: BlockingRecv error: recv got EOF from 127.23.56.65:57733 (error 108)
Metrics: {"server-negotiator.queue_time_us":12,"thread_start_us":116,"threads_started":1}
W20260810 23:03:50.784927 1330 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:03:50.858533 1334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:50.858744 1334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:50.858778 1334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:50.860023 1334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:50.860085 1334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:50.860132 1334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:03:50.861562 1334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.66.keytab
--principal=oryx/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:34149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=38367
--tserver_master_addrs=127.23.56.126:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1334
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:50.861812 1334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:50.862033 1334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:50.866981 1341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.867002 1334 server_base.cc:1034] running on GCE node
W20260810 23:03:50.867087 1340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:50.867089 1346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:50.867408 1334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:50.867599 1334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:50.868727 1334 hybrid_clock.cc:648] HybridClock initialized: now 1786403030868704 us; error 40 us; skew 500 ppm
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403030, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:50.871182 1334 init.cc:377] Logged in from keytab as oryx/127.23.56.66@KRBTEST.COM (short username oryx)
I20260810 23:03:50.871663 1334 webserver.cc:460] Webserver started at http://127.23.56.66:38367/ using document root <none> and password file <none>
I20260810 23:03:50.871930 1334 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:50.872020 1334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:50.873292 1334 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:50.873881 1355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.874051 1334 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.874147 1334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "200c805a3b0049dfb0c416c1610f6631"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "3aa0ee3606c2b642e16ef5bfa3bc9f37"
server_key_iv: "9097bae55789bc1ef72a6c57c2ca9b3f"
server_key_version: "encryptionkey@0"
I20260810 23:03:50.874472 1334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:50.899390 1334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:50.901676 1334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:50.901841 1334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:50.902081 1334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:50.902380 1334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:50.902436 1334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.902477 1334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:50.902513 1334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:50.908401 1334 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:34149
I20260810 23:03:50.908455 1484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:34149 every 8 connection(s)
I20260810 23:03:50.908777 1334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:03:50.909991 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 1334
I20260810 23:03:50.910293 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:44273
--local_ip_for_outbound_sockets=127.23.56.67
--tserver_master_addrs=127.23.56.126:33467
--webserver_port=37807
--webserver_interface=127.23.56.67
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.56.67
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:50.916931 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.909468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:57885)
0810 23:03:50.909622 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:50.909663 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:50.910183 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:50.910476 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:50.910481 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:50.911066 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:50.911272 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.911276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.912213 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.912215 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.913394 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.913396 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.913918 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.913919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.914022 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.914517 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:50.914525 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:50.916797 (+ 2272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
W20260810 23:03:50.917003 1333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:50.909484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:57885 (local address 127.23.56.126:33467)
0810 23:03:50.909560 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.909562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.909754 (+ 192us) server_negotiation.cc:407] Connection header received
0810 23:03:50.910288 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.910290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.910308 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.910386 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.911363 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.912120 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.913489 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.913829 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.913902 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.916921 (+ 3019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:57885: BlockingRecv error: recv got EOF from 127.23.56.66:57885 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 23:03:50.917184 1485 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:33467 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.56.126:33467: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:50.921782 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.917412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:46415)
0810 23:03:50.917587 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:50.917597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:50.917645 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:50.917790 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:50.917790 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:50.917922 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:50.917960 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.917961 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.918562 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.918563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.919070 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.919071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.919440 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.919441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.919503 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.919921 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:50.919925 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:50.921706 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
W20260810 23:03:50.921834 1333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:50.917465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:46415 (local address 127.23.56.126:33467)
0810 23:03:50.917533 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.917535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.917633 (+ 98us) server_negotiation.cc:407] Connection header received
0810 23:03:50.917702 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.917703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.917713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.917785 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.918042 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.918478 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.919184 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.919369 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.919443 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.921772 (+ 2329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:46415: BlockingRecv error: recv got EOF from 127.23.56.66:46415 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:50 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:50.926190 1488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:50.922044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:39337)
0810 23:03:50.922094 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:50.922104 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:50.922140 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:50.922267 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:50.922267 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:50.922367 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:50.922391 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.922391 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.923314 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.923315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.923762 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:50.923763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.924203 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.924204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:50.924238 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.924616 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:50.924620 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:50.926151 (+ 1531us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 23:03:50.926306 1333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:50.922092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:39337 (local address 127.23.56.126:33467)
0810 23:03:50.922153 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:50.922154 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:50.922163 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:50.922186 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:50.922187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:50.922196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:50.922257 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:50.922455 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.923223 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.923844 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:50.924136 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:50.924199 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:50.926249 (+ 2050us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:39337: BlockingRecv error: recv got EOF from 127.23.56.66:39337 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 23:03:50.926314 1485 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:03:50.995628 1490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:03:50.995831 1490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:03:50.995853 1490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:03:50.997033 1490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:03:50.997071 1490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:03:50.997097 1490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:03:50.998561 1490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/krb5kdc/oryx_127.23.56.67.keytab
--principal=oryx/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:44273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=37807
--tserver_master_addrs=127.23.56.126:33467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:03:50.998767 1490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:03:50.998973 1490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:03:51.003921 1499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:51.003926 1500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:03:51.003943 1502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:03:51.004184 1490 server_base.cc:1034] running on GCE node
I20260810 23:03:51.004372 1490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:03:51.004567 1490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:03:51.005743 1490 hybrid_clock.cc:648] HybridClock initialized: now 1786403031005716 us; error 35 us; skew 500 ppm
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403031, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:03:51.008263 1490 init.cc:377] Logged in from keytab as oryx/127.23.56.67@KRBTEST.COM (short username oryx)
I20260810 23:03:51.008704 1490 webserver.cc:460] Webserver started at http://127.23.56.67:37807/ using document root <none> and password file <none>
I20260810 23:03:51.008880 1490 fs_manager.cc:359] Metadata directory not provided
I20260810 23:03:51.008931 1490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:03:51.010201 1490 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:03:51.010777 1509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:51.010944 1490 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:03:51.011042 1490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "5a60bef2176749909241627ea4f98acc"
format_stamp: "Formatted at 2026-08-10 23:03:50 on dist-test-slave-1xrh"
server_key: "e8153dc8626bfe2e9bb00d39a99a9e77"
server_key_iv: "25261bd7df583be91117be52a111e6ae"
server_key_version: "encryptionkey@0"
I20260810 23:03:51.011443 1490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:03:51.021258 1490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:03:51.023526 1490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:03:51.023681 1490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:03:51.023907 1490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:03:51.024204 1490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:03:51.024255 1490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:51.024294 1490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:03:51.024322 1490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:03:51.030197 1490 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:44273
I20260810 23:03:51.030249 1645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:44273 every 8 connection(s)
I20260810 23:03:51.030620 1490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:03:51.036312 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 1490
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
W20260810 23:03:51.037984 1333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:51.031388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:49391 (local address 127.23.56.126:33467)
0810 23:03:51.031438 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:51.031440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:51.031649 (+ 209us) server_negotiation.cc:407] Connection header received
0810 23:03:51.032170 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:51.032172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:51.032185 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:51.032248 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:51.033146 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.033646 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.034586 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.034789 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.034872 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.037938 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:49391: BlockingRecv error: recv got EOF from 127.23.56.67:49391 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:03:51.038084 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:51.031351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:49391)
0810 23:03:51.031527 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:51.031565 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:51.032063 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:51.032314 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:51.032317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:51.032854 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:51.033042 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.033046 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.033745 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.033746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.034497 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.034499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.034883 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.034884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.035010 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.035529 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:51.035536 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:51.037868 (+ 2332us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
W20260810 23:03:51.038297 1646 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:33467 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.56.126:33467: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:51.042739 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:51.038550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:50867)
0810 23:03:51.038592 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:51.038601 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:51.038630 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:51.038757 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:51.038758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:51.038894 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:51.038939 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.038941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.039562 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.039563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.040072 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.040073 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.040415 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.040416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.040486 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.040890 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:51.040894 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:51.042661 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":19}
W20260810 23:03:51.042790 1333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:51.038586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:50867 (local address 127.23.56.126:33467)
0810 23:03:51.038621 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:51.038623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:51.038631 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:03:51.038682 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:51.038683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:51.038689 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:51.038733 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:51.039009 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.039477 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.040156 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.040334 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.040415 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.042728 (+ 2313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:50867: BlockingRecv error: recv got EOF from 127.23.56.67:50867 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:03:51.044521 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:51.036858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48782 (local address 127.23.56.126:33467)
0810 23:03:51.036920 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:51.036921 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:51.036992 (+ 71us) server_negotiation.cc:407] Connection header received
0810 23:03:51.037071 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:51.037071 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:51.037081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:51.037139 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:51.037763 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.038472 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.039529 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.039739 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.039811 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.041183 (+ 1372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:03:51.041187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:03:51.041187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:03:51.041199 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:03:51.043089 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:51.043524 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:51.043528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:51.043528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:51.043643 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:03:51.043857 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:03:51.043858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:03:51.043858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:03:51.044061 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:03:51.044187 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 23:03:51.044307 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 23:03:51.044414 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:51.047410 1648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:51.043054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:51021)
0810 23:03:51.043165 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:51.043176 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:51.043211 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:51.043338 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:51.043338 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:51.043438 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:51.043460 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.043461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.044212 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.044213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.044933 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.044936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.045367 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.045368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.045424 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.045754 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:51.045757 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:51.047368 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94}
W20260810 23:03:51.047502 1333 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:51.043099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:51021 (local address 127.23.56.126:33467)
0810 23:03:51.043186 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:51.043188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:51.043199 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:03:51.043269 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:51.043270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:51.043279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:51.043335 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:51.043563 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.044133 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.045025 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.045296 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.045360 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.047426 (+ 2066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:51021: BlockingRecv error: recv got EOF from 127.23.56.67:51021 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 23:03:51.047529 1646 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:51.789923 1650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:51.785310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:39021)
0810 23:03:51.785456 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:51.785466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:51.785535 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:51.785670 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:51.785671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:51.785837 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:51.785887 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.785889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.786567 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.786568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.787115 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.787116 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.787501 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.787502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.787546 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.788008 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:51.788013 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:51.789816 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
W20260810 23:03:51.789974 1651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:51.785356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:39021 (local address 127.23.56.126:33467)
0810 23:03:51.785491 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:51.785493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:51.785521 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:03:51.785581 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:51.785581 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:51.785593 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:51.785651 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:51.785951 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.786436 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.787234 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.787426 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.787486 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.789900 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:39021: BlockingRecv error: recv got EOF from 127.23.56.65:39021 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:51 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:51.931533 1652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:51.926632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:40765)
0810 23:03:51.926804 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:51.926816 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:51.926909 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:51.927076 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:51.927077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:51.927275 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:51.927344 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.927346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.928109 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.928111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.928652 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:51.928653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.929045 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.929046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:51.929122 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.929577 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:51.929581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:51.931426 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":94,"threads_started":1}
W20260810 23:03:51.931573 1651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:51.926708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:40765 (local address 127.23.56.126:33467)
0810 23:03:51.926801 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:51.926804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:51.926890 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:03:51.926984 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:51.926985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:51.927013 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:51.927061 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:51.927424 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.928023 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.928741 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:51.928951 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:51.929042 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:51.931512 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:40765: BlockingRecv error: recv got EOF from 127.23.56.66:40765 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 23:03:52 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:52 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:52.052505 1653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:52.047899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:37563)
0810 23:03:52.048080 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:52.048094 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:52.048163 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:52.048316 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:52.048317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:52.048532 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:52.048587 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.048588 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.049309 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.049310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.049839 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.049840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.050166 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.050167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.050204 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.050707 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:52.050712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:52.052437 (+ 1725us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
W20260810 23:03:52.052637 1651 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:52.047935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:37563 (local address 127.23.56.126:33467)
0810 23:03:52.048029 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:52.048032 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:52.048169 (+ 137us) server_negotiation.cc:407] Connection header received
0810 23:03:52.048206 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:52.048207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:52.048218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:52.048269 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:52.048682 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.049235 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.049916 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.050101 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.050153 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.052534 (+ 2381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:37563: BlockingRecv error: recv got EOF from 127.23.56.67:37563 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 23:03:52 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:52 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:52.794953 1654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:52.790424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:40791)
0810 23:03:52.790569 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:52.790584 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:52.790649 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:52.790780 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:52.790781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:52.790948 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:52.791003 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.791004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.791681 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.791681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.792217 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.792219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.792551 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.792552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.792594 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.793125 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:52.793130 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:52.794894 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":68,"threads_started":1}
W20260810 23:03:52.795074 1655 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:52.790481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:40791 (local address 127.23.56.126:33467)
0810 23:03:52.790639 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:52.790641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:52.790649 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:03:52.790695 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:52.790696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:52.790707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:52.790763 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:52.791071 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.791616 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.792287 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.792474 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.792538 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.794988 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:40791: BlockingRecv error: recv got EOF from 127.23.56.65:40791 (error 108)
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":94,"threads_started":1}
Aug 10 23:03:52 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:52 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:52.936246 1657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:52.931977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:58493)
0810 23:03:52.932119 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:52.932130 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:52.932184 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:52.932308 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:52.932309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:52.932468 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:52.932522 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.932524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.933136 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.933137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.933682 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:52.933683 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.933983 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.933984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:52.934027 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.934439 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:52.934443 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:52.936190 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
W20260810 23:03:52.936354 1655 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:52.932068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:58493 (local address 127.23.56.126:33467)
0810 23:03:52.932134 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:52.932135 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:52.932171 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:03:52.932228 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:52.932228 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:52.932238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:52.932290 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:52.932588 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.933072 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.933749 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:52.933922 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:52.933967 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:52.936283 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:58493: BlockingRecv error: recv got EOF from 127.23.56.66:58493 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Aug 10 23:03:53 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:53 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:53.057330 1658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:53.052975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:47065)
0810 23:03:53.053145 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:53.053160 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:53.053221 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:53.053354 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:53.053355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:53.053519 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:53.053568 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.053570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.054171 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.054171 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.054683 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.054684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.055012 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.055013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.055053 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.055562 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:53.055567 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:53.057270 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
W20260810 23:03:53.057451 1655 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:53.053121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:47065 (local address 127.23.56.126:33467)
0810 23:03:53.053190 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:53.053192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:53.053204 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:03:53.053278 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:53.053279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:53.053292 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:53.053331 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:53.053630 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.054108 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.054759 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.054942 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.054994 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.057371 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:47065: BlockingRecv error: recv got EOF from 127.23.56.67:47065 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 23:03:53 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:53 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:53.799966 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:53.795435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:57137)
0810 23:03:53.795596 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:53.795607 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:53.795679 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:53.795804 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:53.795805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:53.795971 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:53.796024 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.796025 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.796639 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.796639 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.797159 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.797161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.797485 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.797486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.797526 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.797966 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:53.797970 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:53.799898 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":93,"threads_started":1}
W20260810 23:03:53.800071 1661 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:53.795510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:57137 (local address 127.23.56.126:33467)
0810 23:03:53.795623 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:53.795625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:53.795667 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:03:53.795722 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:53.795722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:53.795732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:53.795786 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:53.796090 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.796575 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.797233 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.797419 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.797467 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.799992 (+ 2525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:57137: BlockingRecv error: recv got EOF from 127.23.56.65:57137 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
Aug 10 23:03:53 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:53 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:53.941512 1662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:53.936812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:58555)
0810 23:03:53.937002 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:53.937013 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:53.937098 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:53.937232 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:53.937233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:53.937477 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:53.937529 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.937531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.938207 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.938208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.938733 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:53.938735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.939104 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.939105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:53.939180 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.939702 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:53.939708 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:53.941440 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":93,"threads_started":1}
W20260810 23:03:53.941622 1661 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:53.936900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:58555 (local address 127.23.56.126:33467)
0810 23:03:53.936975 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:53.936977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:53.937077 (+ 100us) server_negotiation.cc:407] Connection header received
0810 23:03:53.937163 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:53.937164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:53.937177 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:53.937227 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:53.937606 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.938106 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.938811 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:53.939014 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:53.939106 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:53.941551 (+ 2445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:58555: BlockingRecv error: recv got EOF from 127.23.56.66:58555 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 23:03:54 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:54 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:54.062863 1663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:54.057961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:56107)
0810 23:03:54.058161 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:54.058172 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:54.058262 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:54.058422 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:54.058423 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:54.058618 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:54.058666 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:54.058668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.059412 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.059413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:54.059933 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:54.059934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.060306 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.060307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:54.060374 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:54.060836 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:54.060840 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:54.062764 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":114,"threads_started":1}
W20260810 23:03:54.062929 1661 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:54.058050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:56107 (local address 127.23.56.126:33467)
0810 23:03:54.058139 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:54.058141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:54.058244 (+ 103us) server_negotiation.cc:407] Connection header received
0810 23:03:54.058320 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:54.058321 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:54.058336 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:54.058413 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:54.058754 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.059330 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.060027 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.060233 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.060308 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:54.062868 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:56107: BlockingRecv error: recv got EOF from 127.23.56.67:56107 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 23:03:54 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:54 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:54.805328 1665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:54.800483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:45183)
0810 23:03:54.800678 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:54.800697 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:54.800756 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:54.800968 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:54.800969 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:54.801136 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:54.801187 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:54.801189 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.801820 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.801821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:54.802337 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:54.802338 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.802667 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.802668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:54.802709 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:54.803398 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:54.803405 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:54.805235 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":62,"threads_started":1}
W20260810 23:03:54.805421 1666 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:54.800694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:45183 (local address 127.23.56.126:33467)
0810 23:03:54.800836 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:54.800838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:54.800847 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:54.800877 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:54.800878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:54.800888 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:54.800947 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:54.801259 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.801755 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.802409 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.802604 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.802652 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:54.805345 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:45183: BlockingRecv error: recv got EOF from 127.23.56.65:45183 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
Aug 10 23:03:54 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:54 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:54.947067 1667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:54.942090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:48965)
0810 23:03:54.942253 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:54.942268 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:54.942352 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:54.942497 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:54.942498 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:54.942675 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:54.942730 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:54.942732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.943524 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.943526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:54.944070 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:54.944072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.944454 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.944455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:54.944514 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:54.945015 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:54.945020 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:54.946960 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":95,"threads_started":1}
W20260810 23:03:54.947098 1666 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:54.942130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:48965 (local address 127.23.56.126:33467)
0810 23:03:54.942176 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:54.942178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:54.942349 (+ 171us) server_negotiation.cc:407] Connection header received
0810 23:03:54.942410 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:54.942411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:54.942422 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:54.942477 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:54.942795 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.943426 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.944161 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:54.944368 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:54.944442 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:54.947038 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:48965: BlockingRecv error: recv got EOF from 127.23.56.66:48965 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 10 23:03:55 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:55 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:55.068413 1668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.063404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:34995)
0810 23:03:55.063629 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.063641 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:55.063719 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:55.063930 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:55.063932 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:55.064133 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:55.064180 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.064182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.064880 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.064882 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.065433 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.065434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.065840 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.065841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.065943 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.066442 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:55.066447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:55.068355 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":119,"threads_started":1}
W20260810 23:03:55.068547 1666 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:55.063540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:34995 (local address 127.23.56.126:33467)
0810 23:03:55.063622 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.063625 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.063695 (+ 70us) server_negotiation.cc:407] Connection header received
0810 23:03:55.063834 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.063836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.063850 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.063922 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:55.064281 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.064788 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.065532 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.065742 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.065837 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.068459 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:34995: BlockingRecv error: recv got EOF from 127.23.56.67:34995 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Aug 10 23:03:55 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:55 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:55.816357 1669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.805825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:43727)
0810 23:03:55.805963 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.805974 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:55.806037 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:55.806170 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:55.806171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:55.806337 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:55.806386 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.806387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.807009 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.807010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.807554 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.807556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.807901 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.807903 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.807961 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.814281 (+ 6320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:55.814287 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:55.816265 (+ 1978us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
W20260810 23:03:55.816443 1670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:55.805875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:43727 (local address 127.23.56.126:33467)
0810 23:03:55.806022 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.806024 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.806036 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:03:55.806090 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.806091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.806101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.806152 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:55.806450 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.806942 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.807618 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.807803 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.807855 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.816363 (+ 8508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:43727: BlockingRecv error: recv got EOF from 127.23.56.65:43727 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
Aug 10 23:03:55 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:55 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:55.952309 1673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:55.947704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:35235)
0810 23:03:55.947820 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:55.947831 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:55.947904 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:55.948067 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:55.948068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:55.948313 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:55.948410 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.948412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.949161 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.949163 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.949701 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:55.949703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.950034 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.950035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:55.950075 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.950509 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:55.950513 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:55.952232 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":64,"threads_started":1}
W20260810 23:03:55.952441 1670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:55.947780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:35235 (local address 127.23.56.126:33467)
0810 23:03:55.947828 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:55.947830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:55.947883 (+ 53us) server_negotiation.cc:407] Connection header received
0810 23:03:55.947960 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:55.947961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:55.947975 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:55.948058 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:55.948470 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.948970 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.949779 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:55.949959 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:55.950014 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:55.952362 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:35235: BlockingRecv error: recv got EOF from 127.23.56.66:35235 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Aug 10 23:03:56 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:56 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:56.073443 1674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:56.069020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:51581)
0810 23:03:56.069199 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:56.069210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:56.069268 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:56.069400 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:56.069401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:56.069575 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:56.069626 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:56.069627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.070247 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.070247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:56.070763 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:56.070764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.071094 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.071095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:56.071158 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:56.071595 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:56.071600 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:56.073383 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
W20260810 23:03:56.073563 1670 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:56.069096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:51581 (local address 127.23.56.126:33467)
0810 23:03:56.069165 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:56.069167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:56.069268 (+ 101us) server_negotiation.cc:407] Connection header received
0810 23:03:56.069323 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:56.069324 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:56.069336 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:56.069382 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:56.069689 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.070183 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.070834 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.071031 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.071080 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:56.073477 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:51581: BlockingRecv error: recv got EOF from 127.23.56.67:51581 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Aug 10 23:03:56 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:56 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:56.821611 1675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:56.816859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:44285)
0810 23:03:56.817039 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:56.817049 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:56.817106 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:56.817240 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:56.817241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:56.817413 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:56.817463 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:56.817465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.818088 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.818089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:56.818616 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:56.818618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.818939 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.818939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:56.818977 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:56.819535 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:56.819545 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:56.821512 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":111,"threads_started":1}
W20260810 23:03:56.821671 1677 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:56.816930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:44285 (local address 127.23.56.126:33467)
0810 23:03:56.817062 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:56.817064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:56.817093 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:03:56.817163 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:56.817164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:56.817175 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:56.817224 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:56.817530 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.818026 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.818685 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.818877 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.818924 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:56.821599 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:44285: BlockingRecv error: recv got EOF from 127.23.56.65:44285 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":53,"threads_started":1}
Aug 10 23:03:56 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:56 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:56.957154 1678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:56.952713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:42705)
0810 23:03:56.952857 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:56.952868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:56.952929 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:56.953069 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:56.953069 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:56.953226 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:56.953276 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:56.953277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.953896 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.953896 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:56.954387 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:56.954388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.954698 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.954699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:56.954738 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:56.955191 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:56.955196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:56.957057 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
W20260810 23:03:56.957206 1677 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:56.952879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:42705 (local address 127.23.56.126:33467)
0810 23:03:56.952946 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:56.952948 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:56.952957 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:03:56.952988 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:56.952989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:56.953000 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:56.953051 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:56.953339 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.953834 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.954455 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:56.954637 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:56.954684 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:56.957141 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:42705: BlockingRecv error: recv got EOF from 127.23.56.66:42705 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Aug 10 23:03:57 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:57 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:57.078397 1679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:57.073934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:34361)
0810 23:03:57.074074 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:57.074084 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:57.074140 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:57.074272 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:57.074273 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:57.074428 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:57.074476 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:57.074477 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.075092 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.075093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:57.075630 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:57.075631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.075989 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.075990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:57.076028 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:57.076501 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:57.076505 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:57.078338 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
W20260810 23:03:57.078532 1677 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:57.074026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:34361 (local address 127.23.56.126:33467)
0810 23:03:57.074072 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:57.074074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:57.074128 (+ 54us) server_negotiation.cc:407] Connection header received
0810 23:03:57.074199 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:57.074200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:57.074211 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:57.074252 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:57.074543 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.075027 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.075719 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.075909 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.075977 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:57.078443 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:34361: BlockingRecv error: recv got EOF from 127.23.56.67:34361 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Aug 10 23:03:57 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:57 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:57.826668 1680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:57.822161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:33469)
0810 23:03:57.822378 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:57.822391 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:57.822456 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:57.822591 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:57.822591 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:57.822757 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:57.822804 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:57.822806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.823439 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.823440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:57.823980 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:57.823982 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.824308 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.824309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:57.824352 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:57.824804 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:57.824809 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:57.826571 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":75,"threads_started":1}
W20260810 23:03:57.826725 1681 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:57.822196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:33469 (local address 127.23.56.126:33467)
0810 23:03:57.822320 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:57.822322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:57.822460 (+ 138us) server_negotiation.cc:407] Connection header received
0810 23:03:57.822514 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:57.822514 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:57.822526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:57.822578 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:57.822861 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.823374 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.824056 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.824244 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.824292 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:57.826652 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:33469: BlockingRecv error: recv got EOF from 127.23.56.65:33469 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
Aug 10 23:03:57 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:57 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:57.962446 1682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:57.957648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:39017)
0810 23:03:57.957827 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:57.957838 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:57.957899 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:57.958027 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:57.958028 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:57.958197 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:57.958250 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:57.958251 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.958954 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.958954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:57.959502 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:57.959503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.959822 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.959823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:57.959862 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:57.960300 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:57.960304 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:57.962372 (+ 2068us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":111,"threads_started":1}
W20260810 23:03:57.962570 1681 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:57.957704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:39017 (local address 127.23.56.126:33467)
0810 23:03:57.957781 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:57.957784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:57.957899 (+ 115us) server_negotiation.cc:407] Connection header received
0810 23:03:57.957948 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:57.957949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:57.957960 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:57.958008 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:57.958370 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.958879 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.959574 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:57.959753 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:57.959805 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:57.962470 (+ 2665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:39017: BlockingRecv error: recv got EOF from 127.23.56.66:39017 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 23:03:58 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:58 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:58.083665 1683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.078881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:53317)
0810 23:03:58.079054 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.079067 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.079184 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.079357 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.079358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.079548 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.079610 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.079612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.080294 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.080296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.080838 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.080840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.081256 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.081257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.081320 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.081768 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.081773 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.083607 (+ 1834us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":88,"threads_started":1}
W20260810 23:03:58.083847 1681 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:58.078957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:53317 (local address 127.23.56.126:33467)
0810 23:03:58.079040 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.079043 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.079148 (+ 105us) server_negotiation.cc:407] Connection header received
0810 23:03:58.079261 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.079262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.079277 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.079350 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.079709 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.080209 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.080927 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.081181 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.081256 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.083764 (+ 2508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:53317: BlockingRecv error: recv got EOF from 127.23.56.67:53317 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 23:03:58 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:58 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:58.831467 1684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.827163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:49037)
0810 23:03:58.827313 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.827323 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.827373 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.827520 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.827521 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.827686 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.827735 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.827737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.828364 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.828365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.828878 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.828879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.829230 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.829231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.829271 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.829711 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.829715 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.831409 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":82,"threads_started":1}
W20260810 23:03:58.831596 1685 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:58.827285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:49037 (local address 127.23.56.126:33467)
0810 23:03:58.827404 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.827406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.827414 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:03:58.827445 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.827446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.827456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.827516 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.827797 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.828298 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.828950 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.829165 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.829215 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.831509 (+ 2294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:49037: BlockingRecv error: recv got EOF from 127.23.56.65:49037 (error 108)
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
Aug 10 23:03:58 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:58 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:58.967751 1686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:58.962886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:58059)
0810 23:03:58.963042 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:58.963056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:58.963144 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:58.963291 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:58.963291 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:58.963455 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:58.963518 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.963520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.964285 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.964285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.964875 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:58.964878 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.965391 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.965392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:58.965459 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.965923 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:58.965928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:58.967693 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":74,"threads_started":1}
W20260810 23:03:58.967880 1685 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:58.963031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:58059 (local address 127.23.56.126:33467)
0810 23:03:58.963096 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:58.963097 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:58.963112 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:03:58.963210 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:58.963211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:58.963222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:58.963269 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:58.963580 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.964219 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.964975 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:58.965315 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:58.965377 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:58.967801 (+ 2424us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:58059: BlockingRecv error: recv got EOF from 127.23.56.66:58059 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 23:03:59 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:59 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:59.088784 1687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:59.084245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:34891)
0810 23:03:59.084405 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:59.084415 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:59.084477 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:59.084612 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:59.084613 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:59.084778 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:59.084827 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:59.084828 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.085439 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.085440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:59.085998 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:59.086000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.086389 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.086390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:59.086431 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:59.086917 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:59.086921 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:59.088722 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
W20260810 23:03:59.088894 1685 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:59.084289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:34891 (local address 127.23.56.126:33467)
0810 23:03:59.084352 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:59.084354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:59.084476 (+ 122us) server_negotiation.cc:407] Connection header received
0810 23:03:59.084533 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:59.084533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:59.084543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:59.084593 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:59.084885 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.085375 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.086107 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.086332 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.086397 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:59.088818 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:34891: BlockingRecv error: recv got EOF from 127.23.56.67:34891 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 23:03:59 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:59 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:59.836414 1688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:59.831930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:39069)
0810 23:03:59.832084 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:59.832095 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:59.832146 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:59.832281 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:59.832282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:59.832450 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:59.832502 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:59.832503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.833125 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.833126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:59.833633 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:59.833635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.833955 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.833956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:59.834003 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:59.834440 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:59.834444 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:59.836318 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
W20260810 23:03:59.836473 1690 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:59.832032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:39069 (local address 127.23.56.126:33467)
0810 23:03:59.832160 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:59.832162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:59.832183 (+ 21us) server_negotiation.cc:407] Connection header received
0810 23:03:59.832204 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:59.832205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:59.832215 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:59.832265 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:59.832569 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.833059 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.833703 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.833895 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.833944 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:59.836400 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:39069: BlockingRecv error: recv got EOF from 127.23.56.65:39069 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
Aug 10 23:03:59 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:03:59 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:03:59.973551 1691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:03:59.968355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:50269)
0810 23:03:59.968582 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 23:03:59.968592 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:03:59.968649 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:03:59.968777 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:03:59.968777 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:03:59.968946 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:03:59.969001 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:59.969002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.969658 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.969659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:59.970396 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:03:59.970398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.970712 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.970713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:03:59.970748 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:59.971259 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:03:59.971270 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:03:59.973450 (+ 2180us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":143,"threads_started":1}
W20260810 23:03:59.973585 1690 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:03:59.968464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:50269 (local address 127.23.56.126:33467)
0810 23:03:59.968533 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:03:59.968535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:03:59.968637 (+ 102us) server_negotiation.cc:407] Connection header received
0810 23:03:59.968695 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:03:59.968696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:03:59.968706 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:03:59.968761 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:03:59.969065 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.969557 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.970467 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:03:59.970651 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:03:59.970702 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:03:59.973526 (+ 2824us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:50269: BlockingRecv error: recv got EOF from 127.23.56.66:50269 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Aug 10 23:04:00 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:00 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:00.094069 1692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:00.089286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:37545)
0810 23:04:00.089477 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:00.089503 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:00.089590 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:00.089753 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:00.089755 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:00.089947 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:00.089995 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:00.089996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.090669 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.090670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:00.091274 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:00.091276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.091652 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.091653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:00.091708 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:00.092170 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:00.092175 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:00.094010 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":109,"threads_started":1}
W20260810 23:04:00.094240 1690 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:00.089369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:37545 (local address 127.23.56.126:33467)
0810 23:04:00.089439 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:00.089441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:00.089570 (+ 129us) server_negotiation.cc:407] Connection header received
0810 23:04:00.089653 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:00.089654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:00.089669 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:00.089745 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:00.090081 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.090579 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.091367 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.091566 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.091651 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:00.094112 (+ 2461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:37545: BlockingRecv error: recv got EOF from 127.23.56.67:37545 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Aug 10 23:04:00 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:00 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:00.845762 1693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:00.836901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:50785)
0810 23:04:00.837039 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:00.837050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:00.837102 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:00.837269 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:00.837270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:00.837435 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:00.837485 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:00.837487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.838106 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.838108 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:00.838619 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:00.838620 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.838938 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.838939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:00.838978 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:00.843829 (+ 4851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:00.843834 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:00.845653 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260810 23:04:00.845837 1695 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:00.836972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:50785 (local address 127.23.56.126:33467)
0810 23:04:00.837121 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:00.837124 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:00.837132 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:04:00.837162 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:00.837163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:00.837174 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:00.837264 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:00.837548 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.838041 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.838689 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.838878 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.838926 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:00.845740 (+ 6814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:50785: BlockingRecv error: recv got EOF from 127.23.56.65:50785 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":92,"threads_started":1}
Aug 10 23:04:00 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:00 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:00.979008 1696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:00.974131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:58407)
0810 23:04:00.974333 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:00.974343 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:00.974400 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:00.974575 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:00.974576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:00.974743 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:00.974793 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:00.974795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.975523 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.975524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:00.976238 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:00.976241 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.976563 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.976563 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:00.976604 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:00.977048 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:00.977053 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:00.978906 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":120,"threads_started":1}
W20260810 23:04:00.979046 1695 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:00.974231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:58407 (local address 127.23.56.126:33467)
0810 23:04:00.974296 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:00.974299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:00.974399 (+ 100us) server_negotiation.cc:407] Connection header received
0810 23:04:00.974477 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:00.974478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:00.974510 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:00.974551 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:00.974874 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.975420 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.976305 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:00.976500 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:00.976553 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:00.978986 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:58407: BlockingRecv error: recv got EOF from 127.23.56.66:58407 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Aug 10 23:04:01 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:01 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:01.099154 1697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:01.094585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:41937)
0810 23:04:01.094730 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:01.094744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:01.094811 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:01.094951 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:01.094952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:01.095179 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:01.095239 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:01.095241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.095901 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.095902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:01.096440 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:01.096441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.096829 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.096830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:01.096887 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:01.097314 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:01.097319 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:01.099036 (+ 1717us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260810 23:04:01.099210 1695 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:01.094670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:41937 (local address 127.23.56.126:33467)
0810 23:04:01.094734 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:01.094736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:01.094800 (+ 64us) server_negotiation.cc:407] Connection header received
0810 23:04:01.094872 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:01.094873 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:01.094887 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:01.094943 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:01.095315 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.095815 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.096547 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.096754 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.096831 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:01.099119 (+ 2288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:41937: BlockingRecv error: recv got EOF from 127.23.56.67:41937 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Aug 10 23:04:01 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:01 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:01.850800 1698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:01.846235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:53641)
0810 23:04:01.846373 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:01.846390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:01.846439 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:01.846650 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:01.846651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:01.846827 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:01.846875 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:01.846877 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.847634 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.847636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:01.848178 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:01.848180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.848514 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.848515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:01.848555 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:01.849024 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:01.849029 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:01.850728 (+ 1699us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":86,"threads_started":1}
W20260810 23:04:01.850936 1699 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:01.846282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:53641 (local address 127.23.56.126:33467)
0810 23:04:01.846413 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:01.846415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:01.846427 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:04:01.846558 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:01.846559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:01.846573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:01.846636 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:01.846940 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.847525 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.848250 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.848442 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.848494 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:01.850826 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:53641: BlockingRecv error: recv got EOF from 127.23.56.65:53641 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
Aug 10 23:04:01 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:01 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:01.984284 1701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:01.979545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:55751)
0810 23:04:01.979717 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:01.979730 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:01.979804 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:01.979969 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:01.979970 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:01.980173 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:01.980219 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:01.980220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.980959 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.980960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:01.981505 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:01.981507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.981878 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.981879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:01.981946 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:01.982387 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:01.982392 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:01.984208 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
W20260810 23:04:01.984349 1699 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:01.979640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:55751 (local address 127.23.56.126:33467)
0810 23:04:01.979717 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:01.979720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:01.979783 (+ 63us) server_negotiation.cc:407] Connection header received
0810 23:04:01.979881 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:01.979882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:01.979892 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:01.979945 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:01.980304 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.980877 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.981602 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:01.981797 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:01.981865 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:01.984302 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:55751: BlockingRecv error: recv got EOF from 127.23.56.66:55751 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 23:04:02 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:02 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:02.104523 1702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:02.099754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:38867)
0810 23:04:02.099934 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:02.099946 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:02.100020 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:02.100154 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:02.100155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:02.100350 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:02.100396 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:02.100398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.101096 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.101097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:02.101620 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:02.101622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.101986 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.101987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:02.102053 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:02.102570 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:02.102575 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:02.104449 (+ 1874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":95,"threads_started":1}
W20260810 23:04:02.104619 1699 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:02.099828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:38867 (local address 127.23.56.126:33467)
0810 23:04:02.099921 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:02.099924 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:02.099992 (+ 68us) server_negotiation.cc:407] Connection header received
0810 23:04:02.100075 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:02.100076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:02.100090 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:02.100135 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:02.100479 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.100999 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.101704 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.101912 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.101988 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:02.104551 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:38867: BlockingRecv error: recv got EOF from 127.23.56.67:38867 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 10 23:04:02 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:02 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:02.855827 1703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:02.851312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:42443)
0810 23:04:02.851439 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:02.851449 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:02.851513 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:02.851653 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:02.851655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:02.851827 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:02.851874 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:02.851876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.852518 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.852518 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:02.853034 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:02.853035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.853356 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.853357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:02.853395 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:02.853851 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:02.853855 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:02.855748 (+ 1893us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":68,"threads_started":1}
W20260810 23:04:02.855880 1704 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:02.851370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:42443 (local address 127.23.56.126:33467)
0810 23:04:02.851523 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:02.851526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:02.851540 (+ 14us) server_negotiation.cc:407] Connection header received
0810 23:04:02.851563 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:02.851564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:02.851574 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:02.851635 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:02.851937 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.852433 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.853102 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.853292 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.853340 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:02.855825 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:42443: BlockingRecv error: recv got EOF from 127.23.56.65:42443 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
Aug 10 23:04:02 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:02 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:02.989305 1705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:02.984761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:48727)
0810 23:04:02.984906 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:02.984916 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:02.984975 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:02.985111 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:02.985112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:02.985282 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:02.985330 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:02.985331 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.985974 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.985975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:02.986499 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:02.986501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.986823 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.986824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:02.986870 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:02.987359 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:02.987364 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:02.989205 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":87,"threads_started":1}
W20260810 23:04:02.989351 1704 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:02.984824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:48727 (local address 127.23.56.126:33467)
0810 23:04:02.984890 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:02.984892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:02.984958 (+ 66us) server_negotiation.cc:407] Connection header received
0810 23:04:02.985031 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:02.985032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:02.985042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:02.985092 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:02.985392 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.985886 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.986567 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:02.986755 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:02.986807 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:02.989287 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:48727: BlockingRecv error: recv got EOF from 127.23.56.66:48727 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 23:04:03 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:03 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:03.109994 1708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.105079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:50619)
0810 23:04:03.105264 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.105276 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.105371 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.105562 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.105563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.105802 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:03.105848 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.105850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.106567 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.106569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.107088 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.107090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.107499 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.107500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.107576 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.108065 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:03.108070 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:03.109895 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":96,"threads_started":1}
W20260810 23:04:03.110054 1704 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:03.105172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:50619 (local address 127.23.56.126:33467)
0810 23:04:03.105258 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.105260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.105352 (+ 92us) server_negotiation.cc:407] Connection header received
0810 23:04:03.105428 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.105429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.105442 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.105523 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:03.105946 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.106471 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.107220 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.107417 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.107497 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.109987 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:50619: BlockingRecv error: recv got EOF from 127.23.56.67:50619 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 23:04:03 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:03 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:03.861164 1709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.856351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:47265)
0810 23:04:03.856532 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.856545 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.856615 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.856767 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.856769 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.856951 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:03.857003 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.857005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.857646 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.857647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.858208 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.858210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.858531 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.858532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.858574 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.859099 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:03.859105 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:03.861093 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":117,"threads_started":1}
W20260810 23:04:03.861266 1710 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:03.856432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:47265 (local address 127.23.56.126:33467)
0810 23:04:03.856587 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.856589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.856599 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:04:03.856679 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.856680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.856690 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.856745 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:03.857071 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.857562 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.858281 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.858466 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.858516 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.861188 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:47265: BlockingRecv error: recv got EOF from 127.23.56.65:47265 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
Aug 10 23:04:03 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:03 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:03.994740 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:03.989806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:33579)
0810 23:04:03.989985 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:03.989997 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:03.990055 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:03.990192 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:03.990192 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:03.990434 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:03.990512 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.990515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.991378 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.991379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.991958 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:03.991960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.992273 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.992274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:03.992320 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.992887 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:03.992892 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:03.994674 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":110,"threads_started":1}
W20260810 23:04:03.994885 1710 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:03.989924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:33579 (local address 127.23.56.126:33467)
0810 23:04:03.990000 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:03.990003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:03.990045 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:04:03.990107 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:03.990108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:03.990118 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:03.990173 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:03.990603 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.991305 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.992033 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:03.992206 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:03.992253 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:03.994801 (+ 2548us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:33579: BlockingRecv error: recv got EOF from 127.23.56.66:33579 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 23:04:04 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:04 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:04.115653 1712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.110494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:49927)
0810 23:04:04.110699 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.110711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.110782 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.110996 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.110998 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.111210 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:04.111262 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.111264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.111909 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.111910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.112475 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.112477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.112848 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.112849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.112902 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.113557 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:04.113562 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:04.115541 (+ 1979us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":120,"threads_started":1}
W20260810 23:04:04.115700 1710 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:04.110582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:49927 (local address 127.23.56.126:33467)
0810 23:04:04.110644 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.110646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.110772 (+ 126us) server_negotiation.cc:407] Connection header received
0810 23:04:04.110828 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.110829 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.110841 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.110890 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:04.111340 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.111840 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.112560 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.112738 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.112786 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.115631 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:49927: BlockingRecv error: recv got EOF from 127.23.56.67:49927 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 23:04:04 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:04 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:04.866560 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.861690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:39243)
0810 23:04:04.861832 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.861844 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.861901 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.862039 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.862040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.862214 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:04.862262 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.862264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.863048 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.863049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.863841 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.863844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.864184 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.864184 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.864223 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.864722 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:04.864726 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:04.866501 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":87,"threads_started":1}
W20260810 23:04:04.866681 1715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:04.861756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:39243 (local address 127.23.56.126:33467)
0810 23:04:04.861893 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.861895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.861905 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:04:04.861957 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.861957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.861968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.862022 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:04.862345 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.862844 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.863921 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.864119 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.864173 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.866598 (+ 2425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:39243: BlockingRecv error: recv got EOF from 127.23.56.65:39243 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":76,"threads_started":1}
Aug 10 23:04:04 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:04 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:05.000020 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:04.995347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:47375)
0810 23:04:04.995559 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:04.995577 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:04.995641 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:04.995773 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:04.995774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:04.995981 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:04.996032 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.996034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.996638 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.996639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.997178 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:04.997180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.997514 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.997515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:04.997556 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:04.998124 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:04.998128 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:04.999954 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":72,"threads_started":1}
W20260810 23:04:05.000160 1715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:04.995480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:47375 (local address 127.23.56.126:33467)
0810 23:04:04.995572 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:04.995574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:04.995623 (+ 49us) server_negotiation.cc:407] Connection header received
0810 23:04:04.995690 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:04.995690 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:04.995701 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:04.995746 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:04.996095 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.996573 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.997243 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:04.997433 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:04.997498 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.000078 (+ 2580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:47375: BlockingRecv error: recv got EOF from 127.23.56.66:47375 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 23:04:05 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:05 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.67@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:05.121098 1717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.116120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.67:58665)
0810 23:04:05.116289 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.116299 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:05.116348 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:05.116514 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:05.116515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:05.116686 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:05.116734 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:05.116735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.117455 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.117456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:05.117978 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:05.117986 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.118373 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.118373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:05.118411 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.118919 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:05.118924 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:05.121037 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":107,"threads_started":1}
W20260810 23:04:05.121196 1715 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:05.116210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:58665 (local address 127.23.56.126:33467)
0810 23:04:05.116331 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:05.116333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:05.116345 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:04:05.116418 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:05.116419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:05.116433 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:05.116496 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:05.116803 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.117374 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.118069 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.118270 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.118363 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.121124 (+ 2761us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.67:58665: BlockingRecv error: recv got EOF from 127.23.56.67:58665 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 23:04:05 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:05 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.65@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:05.876266 1718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:05.867104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.65:37219)
0810 23:04:05.867280 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:05.867293 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:05.867357 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:05.867475 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:05.867476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:05.867648 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:05.867698 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:05.867700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.868398 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.868399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:05.868935 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:05.868937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.869348 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.869350 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:05.869402 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.874183 (+ 4781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:05.874189 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:05.876159 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":112,"threads_started":1}
W20260810 23:04:05.876322 1719 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:05.867204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:37219 (local address 127.23.56.126:33467)
0810 23:04:05.867349 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:05.867351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:05.867360 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:04:05.867399 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:05.867400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:05.867411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:05.867464 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:05.867757 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.868306 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.869041 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:05.869245 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:05.869313 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:05.876247 (+ 6934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.65:37219: BlockingRecv error: recv got EOF from 127.23.56.65:37219 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":68,"threads_started":1}
Aug 10 23:04:06 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
Aug 10 23:04:06 dist-test-slave-1xrh krb5kdc[443](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.56.66@KRBTEST.COM for oryx/127.23.56.126@KRBTEST.COM, Server not found in Kerberos database
I20260810 23:04:06.005159 1721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.000548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33467 (local address 127.23.56.66:55439)
0810 23:04:06.000729 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.000739 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:06.000792 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:06.000945 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:06.000946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:06.001127 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:06.001177 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.001179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.001826 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.001826 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.002339 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.002341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.002680 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.002682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.002734 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.003302 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:06.003308 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:06.005065 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.56.126:33467: Server oryx/127.23.56.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":119,"threads_started":1}
W20260810 23:04:06.005267 1719 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:06.000750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:55439 (local address 127.23.56.126:33467)
0810 23:04:06.000819 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:06.000821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:06.000829 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:04:06.000856 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:06.000857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:06.000868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:06.000923 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:06.001238 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.001718 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.002405 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.002585 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.002633 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.005180 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.56.66:55439: BlockingRecv error: recv got EOF from 127.23.56.66:55439 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:04:06.037532 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 1159
I20260810 23:04:06.046380 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 1334
I20260810 23:04:06.052592 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 1490
I20260810 23:04:06.057567 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 1076
2026-08-10T23:04:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17542 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T23:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:04:06Z Disabled control of system clock
I20260810 23:04:06.088711 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45407
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:45407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 23:04:06.179184 1725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:06.179402 1725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:06.179438 1725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:06.180604 1725 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:06.180639 1725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:06.180653 1725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:06.180667 1725 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:06.180680 1725 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:06.182014 1725 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:45407
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1725
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:06.182214 1725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:06.182471 1725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:06.187958 1731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.187935 1730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.187935 1733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.187964 1725 server_base.cc:1034] running on GCE node
I20260810 23:04:06.188439 1725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:06.188726 1725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:06.189924 1725 hybrid_clock.cc:648] HybridClock initialized: now 1786403046189918 us; error 37 us; skew 500 ppm
I20260810 23:04:06.191262 1725 webserver.cc:460] Webserver started at http://127.23.56.126:46453/ using document root <none> and password file <none>
I20260810 23:04:06.191504 1725 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:06.191577 1725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:06.191740 1725 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:06.192768 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "74cbff34040a4bdb9b5671f70ac2f98e"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "6df06c0114779affc2fefede06403383"
server_key_iv: "01cbe0b1eb4c6ac8efefedb4fc6ecc59"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.193152 1725 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "74cbff34040a4bdb9b5671f70ac2f98e"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "6df06c0114779affc2fefede06403383"
server_key_iv: "01cbe0b1eb4c6ac8efefedb4fc6ecc59"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.194482 1725 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:04:06.195271 1742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.195449 1725 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:04:06.195546 1725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "74cbff34040a4bdb9b5671f70ac2f98e"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "6df06c0114779affc2fefede06403383"
server_key_iv: "01cbe0b1eb4c6ac8efefedb4fc6ecc59"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.195629 1725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:06.222059 1725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:06.222407 1725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:06.222569 1725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:06.226567 1725 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:45407
I20260810 23:04:06.226606 1810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:45407 every 8 connection(s)
I20260810 23:04:06.226961 1725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:06.227680 1811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:06.230270 1811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e: Bootstrap starting.
I20260810 23:04:06.230818 1811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:06.231057 1811 log.cc:824] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:06.231781 1811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e: No bootstrap required, opened a new log
I20260810 23:04:06.233297 1811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cbff34040a4bdb9b5671f70ac2f98e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45407 } }
I20260810 23:04:06.233497 1811 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:06.233556 1811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74cbff34040a4bdb9b5671f70ac2f98e, State: Initialized, Role: FOLLOWER
I20260810 23:04:06.233567 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 1725
I20260810 23:04:06.233659 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:04:06.233714 1811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cbff34040a4bdb9b5671f70ac2f98e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45407 } }
I20260810 23:04:06.233803 1811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:06.233836 1811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:06.233865 23777 external_mini_cluster.cc:1428] Setting key 47da462b3e5db0d5e8d4d4f42c6a19a9
I20260810 23:04:06.233886 1811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:06.234611 1811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cbff34040a4bdb9b5671f70ac2f98e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45407 } }
I20260810 23:04:06.234745 1811 leader_election.cc:302] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74cbff34040a4bdb9b5671f70ac2f98e; no voters:
I20260810 23:04:06.234906 1811 leader_election.cc:288] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:04:06.234948 1817 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:06.235080 1817 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [term 1 LEADER]: Becoming Leader. State: Replica: 74cbff34040a4bdb9b5671f70ac2f98e, State: Running, Role: LEADER
I20260810 23:04:06.235186 1811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:06.235265 1817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cbff34040a4bdb9b5671f70ac2f98e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45407 } }
I20260810 23:04:06.235787 1818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74cbff34040a4bdb9b5671f70ac2f98e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cbff34040a4bdb9b5671f70ac2f98e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45407 } } }
I20260810 23:04:06.235909 1818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:06.236135 1827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:06.236773 1827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:06.236143 1819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74cbff34040a4bdb9b5671f70ac2f98e. Latest consensus state: current_term: 1 leader_uuid: "74cbff34040a4bdb9b5671f70ac2f98e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74cbff34040a4bdb9b5671f70ac2f98e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45407 } } }
I20260810 23:04:06.237531 1819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:06.239558 1827 catalog_manager.cc:1347] Generated new cluster ID: da2648ca4f954e76b7b48cf9f47dd40c
I20260810 23:04:06.239631 1827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:04:06.240419 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.234442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39760 (local address 127.23.56.126:45407)
0810 23:04:06.234654 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:06.234661 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:06.234720 (+ 59us) server_negotiation.cc:407] Connection header received
0810 23:04:06.234901 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:06.234909 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:06.234986 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:06.235474 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:06.235570 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.237406 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.238414 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.238826 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.238915 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.239701 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:06.239706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:06.239707 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:04:06.239908 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:06.239957 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:04:06.239972 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:06.240052 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:06.240140 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 23:04:06.240218 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260810 23:04:06.254634 1827 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:04:06.255275 1827 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:04:06.258674 1827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74cbff34040a4bdb9b5671f70ac2f98e: Generated new TSK 0
I20260810 23:04:06.258867 1827 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:04:06.266299 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45407
--builtin_ntp_servers=127.23.56.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 23:04:06.354215 1838 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:06.354456 1838 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:06.354492 1838 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:06.355674 1838 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:06.355741 1838 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:06.355783 1838 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:04:06.357121 1838 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1838
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:06.357399 1838 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:06.357713 1838 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:06.363610 1849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.363641 1838 server_base.cc:1034] running on GCE node
W20260810 23:04:06.363850 1846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.363720 1847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.364117 1838 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:06.364318 1838 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:06.365494 1838 hybrid_clock.cc:648] HybridClock initialized: now 1786403046365476 us; error 27 us; skew 500 ppm
I20260810 23:04:06.366869 1838 webserver.cc:460] Webserver started at http://127.23.56.65:39535/ using document root <none> and password file <none>
I20260810 23:04:06.367175 1838 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:06.367259 1838 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:06.367451 1838 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:06.368530 1838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "f8ae01affc474182b040a484762a0cd8"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "2448a5a55c3bd41ffb6ff427b6616b37"
server_key_iv: "bda69b62a7bc5086244c5e0111ee7fbd"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.368899 1838 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "f8ae01affc474182b040a484762a0cd8"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "2448a5a55c3bd41ffb6ff427b6616b37"
server_key_iv: "bda69b62a7bc5086244c5e0111ee7fbd"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.370273 1838 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:04:06.371085 1855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.371299 1838 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:04:06.371410 1838 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "f8ae01affc474182b040a484762a0cd8"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "2448a5a55c3bd41ffb6ff427b6616b37"
server_key_iv: "bda69b62a7bc5086244c5e0111ee7fbd"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.371501 1838 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:06.389195 1838 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:06.389498 1838 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:06.389662 1838 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:06.389879 1838 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:06.390209 1838 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:06.390269 1838 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.390308 1838 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:06.390347 1838 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.396356 1838 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:46459
I20260810 23:04:06.396404 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:46459 every 8 connection(s)
I20260810 23:04:06.396714 1838 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:04:06.401857 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.397435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:50055 (local address 127.23.56.126:45407)
0810 23:04:06.397504 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:06.397506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:06.397717 (+ 211us) server_negotiation.cc:407] Connection header received
0810 23:04:06.398315 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:06.398316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:06.398331 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:06.398394 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:06.399070 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.399683 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.400610 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.400806 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.400869 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.401602 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:06.401603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:06.401603 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:04:06.401610 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:06.401624 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:04:06.401628 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:06.401672 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:06.401753 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 23:04:06.401781 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:04:06.402017 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.397385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45407 (local address 127.23.56.65:50055)
0810 23:04:06.397593 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.397630 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:06.398118 (+ 488us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:06.398469 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:06.398473 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:06.398486 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:06.398959 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.398962 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.399769 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.399770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.400526 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.400528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.400886 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.400886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.400988 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.401465 (+ 477us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:04:06.401466 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:06.401503 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:04:06.401524 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:06.401681 (+ 157us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:06.401685 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:06.401687 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:04:06.401738 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 23:04:06.401812 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260810 23:04:06.402289 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 1838
I20260810 23:04:06.402390 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:04:06.402570 1991 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45407
I20260810 23:04:06.402607 23777 external_mini_cluster.cc:1428] Setting key 0e628f8f7611fe35d145de0d9c4b411d
I20260810 23:04:06.402667 1991 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:06.402844 1991 heartbeater.cc:511] Master 127.23.56.126:45407 requested a full tablet report, sending...
I20260810 23:04:06.403457 1765 ts_manager.cc:194] Registered new tserver with Master: f8ae01affc474182b040a484762a0cd8 (127.23.56.65:46459)
I20260810 23:04:06.403628 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45407
--builtin_ntp_servers=127.23.56.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 23:04:06.404194 1765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.65:50055
W20260810 23:04:06.487176 1994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:06.487401 1994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:06.487422 1994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:06.488526 1994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:06.488560 1994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:06.488583 1994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:04:06.489900 1994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:06.490097 1994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:06.490361 1994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:06.495324 1999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.495283 2000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.495292 2002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.495321 1994 server_base.cc:1034] running on GCE node
I20260810 23:04:06.495718 1994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:06.495998 1994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:06.497174 1994 hybrid_clock.cc:648] HybridClock initialized: now 1786403046497165 us; error 33 us; skew 500 ppm
I20260810 23:04:06.498536 1994 webserver.cc:460] Webserver started at http://127.23.56.66:45027/ using document root <none> and password file <none>
I20260810 23:04:06.498781 1994 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:06.498858 1994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:06.499011 1994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:06.500195 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "ee8addaba99046b789b63c0dfd0d85f1"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "f783384ea7cf1921364934d9c01a623c"
server_key_iv: "de27790e6be3ac6347dfe640b5643b66"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.500557 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "ee8addaba99046b789b63c0dfd0d85f1"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "f783384ea7cf1921364934d9c01a623c"
server_key_iv: "de27790e6be3ac6347dfe640b5643b66"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.501971 1994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:04:06.502717 2009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.502893 1994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:04:06.502997 1994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "ee8addaba99046b789b63c0dfd0d85f1"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "f783384ea7cf1921364934d9c01a623c"
server_key_iv: "de27790e6be3ac6347dfe640b5643b66"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.503082 1994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:06.525461 1994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:06.525768 1994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:06.525929 1994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:06.526147 1994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:06.526485 1994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:06.526540 1994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.526582 1994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:06.526608 1994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.532644 1994 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:46317
I20260810 23:04:06.532697 2143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:46317 every 8 connection(s)
I20260810 23:04:06.533107 1994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:04:06.538223 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.533819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:39723 (local address 127.23.56.126:45407)
0810 23:04:06.533863 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:06.533865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:06.534106 (+ 241us) server_negotiation.cc:407] Connection header received
0810 23:04:06.534585 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:06.534586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:06.534597 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:06.534666 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:06.535390 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.535973 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.537019 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.537199 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.537252 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.538003 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:06.538004 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:06.538004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:04:06.538008 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:06.538021 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:04:06.538024 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:06.538078 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:06.538124 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 23:04:06.538165 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:04:06.538460 2147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.533798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45407 (local address 127.23.56.66:39723)
0810 23:04:06.533982 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.534017 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:06.534502 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:06.534716 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:06.534719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:06.534727 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:06.535292 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.535298 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.536068 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.536069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.536941 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.536943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.537273 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.537274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.537381 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.537874 (+ 493us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:04:06.537874 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:06.537901 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:04:06.537923 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:06.538078 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:06.538080 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:06.538082 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:04:06.538143 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 23:04:06.538217 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":92,"threads_started":1}
I20260810 23:04:06.538975 2144 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45407
I20260810 23:04:06.539083 2144 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:06.539305 2144 heartbeater.cc:511] Master 127.23.56.126:45407 requested a full tablet report, sending...
I20260810 23:04:06.539499 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 1994
I20260810 23:04:06.539599 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:04:06.539772 1765 ts_manager.cc:194] Registered new tserver with Master: ee8addaba99046b789b63c0dfd0d85f1 (127.23.56.66:46317)
I20260810 23:04:06.539819 23777 external_mini_cluster.cc:1428] Setting key dda912648de5330b1c631ef3ea304816
I20260810 23:04:06.540354 1765 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.66:39723
I20260810 23:04:06.540848 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45407
--builtin_ntp_servers=127.23.56.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 23:04:06.638371 2148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:06.638571 2148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:06.638592 2148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:06.639744 2148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:06.639781 2148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:06.639804 2148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:04:06.641150 2148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2148
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:06.641374 2148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:06.641654 2148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:06.646564 2153 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.646560 2157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.646576 2154 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.646611 2148 server_base.cc:1034] running on GCE node
I20260810 23:04:06.646912 2148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:06.647198 2148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:06.648353 2148 hybrid_clock.cc:648] HybridClock initialized: now 1786403046648339 us; error 30 us; skew 500 ppm
I20260810 23:04:06.649626 2148 webserver.cc:460] Webserver started at http://127.23.56.67:39487/ using document root <none> and password file <none>
I20260810 23:04:06.649857 2148 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:06.649924 2148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:06.650072 2148 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:06.651086 2148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "958e564bdf164a92a651493fe98399ef"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "42ebb466b892c71ada8573d2099226a8"
server_key_iv: "4053971e97b541ebc69467fc4ee5ecb1"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.651527 2148 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "958e564bdf164a92a651493fe98399ef"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "42ebb466b892c71ada8573d2099226a8"
server_key_iv: "4053971e97b541ebc69467fc4ee5ecb1"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.652820 2148 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:04:06.653506 2164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.653666 2148 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:04:06.653761 2148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "958e564bdf164a92a651493fe98399ef"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "42ebb466b892c71ada8573d2099226a8"
server_key_iv: "4053971e97b541ebc69467fc4ee5ecb1"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.653841 2148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:06.673039 2148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:06.673334 2148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:06.673481 2148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:06.673700 2148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:06.674044 2148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:06.674099 2148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.674134 2148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:06.674162 2148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.680024 2148 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:36885
I20260810 23:04:06.680115 2292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:36885 every 8 connection(s)
I20260810 23:04:06.680433 2148 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:04:06.685652 1816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.681211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:60775 (local address 127.23.56.126:45407)
0810 23:04:06.681280 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:06.681282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:06.681527 (+ 245us) server_negotiation.cc:407] Connection header received
0810 23:04:06.682066 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:06.682068 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:06.682082 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:06.682140 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:06.682800 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.683386 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.684352 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.684559 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.684636 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.685378 (+ 742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:06.685380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:06.685380 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:04:06.685384 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:06.685398 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:04:06.685402 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:06.685450 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:06.685545 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 23:04:06.685573 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:04:06.685875 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.681200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45407 (local address 127.23.56.67:60775)
0810 23:04:06.681387 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:06.681418 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:06.681944 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:06.682203 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:06.682207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:06.682217 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:06.682700 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.682703 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.683475 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.683476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.684256 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:06.684258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:06.684643 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:06.684644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:06.684773 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:06.685239 (+ 466us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:04:06.685240 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:06.685267 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:04:06.685287 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:06.685449 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:06.685454 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:06.685457 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:04:06.685545 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:04:06.685641 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260810 23:04:06.686412 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 2148
I20260810 23:04:06.686487 2293 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45407
I20260810 23:04:06.686522 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:04:06.686585 2293 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:06.686718 23777 external_mini_cluster.cc:1428] Setting key 68c19e4c92b8ed30f0af59f823b80c82
I20260810 23:04:06.686785 2293 heartbeater.cc:511] Master 127.23.56.126:45407 requested a full tablet report, sending...
I20260810 23:04:06.687219 1764 ts_manager.cc:194] Registered new tserver with Master: 958e564bdf164a92a651493fe98399ef (127.23.56.67:36885)
I20260810 23:04:06.687711 1764 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.67:60775
I20260810 23:04:06.687759 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:04:06.689715 1816 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:06.689250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39768 (local address 127.23.56.126:45407)
0810 23:04:06.689318 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:06.689320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:06.689423 (+ 103us) server_negotiation.cc:407] Connection header received
0810 23:04:06.689459 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:06.689460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:06.689470 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:06.689521 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:06.689652 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39768: BlockingRecv error: recv got EOF from 127.0.0.1:39768 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:04:06.689759 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 1838
I20260810 23:04:06.703572 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 1994
I20260810 23:04:06.710425 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 2148
I20260810 23:04:06.718137 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 1725
2026-08-10T23:04:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (659 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T23:04:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:04:06Z Disabled control of system clock
I20260810 23:04:06.747346 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34473
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:34473
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:04:06.829079 2306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:06.829281 2306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:06.829303 2306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:06.830384 2306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:06.830420 2306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:06.830435 2306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:06.830446 2306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:06.830459 2306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:06.831818 2306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:34473
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34473
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:06.832011 2306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:06.832237 2306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:04:06.837045 2306 server_base.cc:1034] running on GCE node
W20260810 23:04:06.837087 2312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.837173 2315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.837328 2313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.837595 2306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:06.837843 2306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:06.838981 2306 hybrid_clock.cc:648] HybridClock initialized: now 1786403046838970 us; error 27 us; skew 500 ppm
I20260810 23:04:06.840274 2306 webserver.cc:460] Webserver started at http://127.23.56.126:35837/ using document root <none> and password file <none>
I20260810 23:04:06.840521 2306 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:06.840569 2306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:06.840719 2306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:06.841847 2306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "722bb254bb594aef8aea78d5595c9b0a"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "51a8e283ca6c8c9aa4e09d9f426183de"
server_key_iv: "f529f9504bced0eed448b0920c570578"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.842206 2306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "722bb254bb594aef8aea78d5595c9b0a"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "51a8e283ca6c8c9aa4e09d9f426183de"
server_key_iv: "f529f9504bced0eed448b0920c570578"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.843654 2306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 23:04:06.844372 2321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.844547 2306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:04:06.844645 2306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "722bb254bb594aef8aea78d5595c9b0a"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "51a8e283ca6c8c9aa4e09d9f426183de"
server_key_iv: "f529f9504bced0eed448b0920c570578"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.844730 2306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:06.848013 2306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:06.848277 2306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:06.848425 2306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:06.852355 2306 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:34473
I20260810 23:04:06.852386 2389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:34473 every 8 connection(s)
I20260810 23:04:06.852701 2306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:06.853307 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:06.855947 2390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a: Bootstrap starting.
I20260810 23:04:06.856508 2390 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:06.856746 2390 log.cc:824] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:06.857347 2390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a: No bootstrap required, opened a new log
I20260810 23:04:06.858815 2390 raft_consensus.cc:348] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722bb254bb594aef8aea78d5595c9b0a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34473 } }
I20260810 23:04:06.858940 2390 raft_consensus.cc:374] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:06.858963 2390 raft_consensus.cc:729] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 722bb254bb594aef8aea78d5595c9b0a, State: Initialized, Role: FOLLOWER
I20260810 23:04:06.859091 2390 consensus_queue.cc:260] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722bb254bb594aef8aea78d5595c9b0a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34473 } }
I20260810 23:04:06.859195 2390 raft_consensus.cc:388] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:06.859231 2390 raft_consensus.cc:482] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:06.859261 2390 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:06.859735 2390 raft_consensus.cc:504] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722bb254bb594aef8aea78d5595c9b0a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34473 } }
I20260810 23:04:06.859817 2390 leader_election.cc:302] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 722bb254bb594aef8aea78d5595c9b0a; no voters:
I20260810 23:04:06.859926 2390 leader_election.cc:288] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:04:06.860024 2393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:06.860172 2390 sys_catalog.cc:564] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:06.860203 2393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [term 1 LEADER]: Becoming Leader. State: Replica: 722bb254bb594aef8aea78d5595c9b0a, State: Running, Role: LEADER
I20260810 23:04:06.860307 2393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722bb254bb594aef8aea78d5595c9b0a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34473 } }
I20260810 23:04:06.860780 2395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 722bb254bb594aef8aea78d5595c9b0a. Latest consensus state: current_term: 1 leader_uuid: "722bb254bb594aef8aea78d5595c9b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722bb254bb594aef8aea78d5595c9b0a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34473 } } }
I20260810 23:04:06.860945 2394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "722bb254bb594aef8aea78d5595c9b0a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "722bb254bb594aef8aea78d5595c9b0a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34473 } } }
I20260810 23:04:06.861021 2394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:06.860945 2395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:06.861398 2402 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:06.861512 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 2306
I20260810 23:04:06.861616 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:04:06.861793 23777 external_mini_cluster.cc:1428] Setting key 7b82c8a9e046a6b08ecab7b5684ba9f4
I20260810 23:04:06.861974 2402 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:06.863999 2402 catalog_manager.cc:1347] Generated new cluster ID: ad845ebadae1416ca4547d5ffc73fc73
I20260810 23:04:06.864058 2402 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:04:06.864624 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:06.862268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.23.56.126:34473)
0810 23:04:06.862520 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:06.862530 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:06.862587 (+ 57us) server_negotiation.cc:407] Connection header received
0810 23:04:06.863026 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:06.863052 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:06.863101 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:06.863257 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:06.864172 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:06.864176 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:06.864176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:04:06.864180 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:06.864230 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:04:06.864245 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:06.864306 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:06.864398 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 23:04:06.864452 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260810 23:04:06.885031 2402 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:04:06.885217 2402 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:04:06.894326 2402 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 722bb254bb594aef8aea78d5595c9b0a: Generated new TSK 0
I20260810 23:04:06.894555 2402 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:04:06.896744 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34473
--builtin_ntp_servers=127.23.56.84:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:04:06.978977 2416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:06.979270 2416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:06.979318 2416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:06.980518 2416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:06.980597 2416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:06.980641 2416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:04:06.981999 2416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:06.982270 2416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:06.982573 2416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:06.988505 2422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.988634 2416 server_base.cc:1034] running on GCE node
W20260810 23:04:06.988580 2425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:06.988519 2421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:06.989040 2416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:06.989250 2416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:06.990417 2416 hybrid_clock.cc:648] HybridClock initialized: now 1786403046990402 us; error 26 us; skew 500 ppm
I20260810 23:04:06.991830 2416 webserver.cc:460] Webserver started at http://127.23.56.65:44635/ using document root <none> and password file <none>
I20260810 23:04:06.992110 2416 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:06.992159 2416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:06.992354 2416 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:06.993531 2416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "e2923a6841ed4373bc38916756befb93"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "5e86ae5355bf247c097016e8fee9bfb8"
server_key_iv: "2e5a3a3649b56152fa013cdb393505fd"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.993940 2416 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "e2923a6841ed4373bc38916756befb93"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "5e86ae5355bf247c097016e8fee9bfb8"
server_key_iv: "2e5a3a3649b56152fa013cdb393505fd"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.995430 2416 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260810 23:04:06.996204 2431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:06.996408 2416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:04:06.996526 2416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "e2923a6841ed4373bc38916756befb93"
format_stamp: "Formatted at 2026-08-10 23:04:06 on dist-test-slave-1xrh"
server_key: "5e86ae5355bf247c097016e8fee9bfb8"
server_key_iv: "2e5a3a3649b56152fa013cdb393505fd"
server_key_version: "encryptionkey@0"
I20260810 23:04:06.996605 2416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:07.000144 2416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:07.000416 2416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:07.000564 2416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:07.000783 2416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:07.001097 2416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:07.001152 2416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:07.001194 2416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:07.001220 2416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:07.007403 2416 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:37263
I20260810 23:04:07.007441 2563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:37263 every 8 connection(s)
I20260810 23:04:07.007777 2416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:04:07.010938 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:07.008568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:34993 (local address 127.23.56.126:34473)
0810 23:04:07.008641 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:07.008643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:07.008899 (+ 256us) server_negotiation.cc:407] Connection header received
0810 23:04:07.009337 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:07.009339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:07.009347 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:07.009413 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:07.010631 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:07.010632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:07.010633 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:04:07.010637 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:07.010653 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:04:07.010657 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:07.010713 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:07.010865 (+ 152us) server_negotiation.cc:299] Negotiation successful
0810 23:04:07.010883 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:04:07.011011 2567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:07.008563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34473 (local address 127.23.56.65:34993)
0810 23:04:07.008764 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:07.008797 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:07.009252 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:07.009490 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:07.009494 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:07.009501 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:07.010475 (+ 974us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:04:07.010476 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:07.010507 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:04:07.010536 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:07.010724 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:07.010730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:07.010732 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:04:07.010793 (+ 61us) client_negotiation.cc:240] Negotiation successful
0810 23:04:07.010843 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260810 23:04:07.011915 2565 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34473
I20260810 23:04:07.012005 2565 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:07.012140 2565 heartbeater.cc:511] Master 127.23.56.126:34473 requested a full tablet report, sending...
I20260810 23:04:07.012356 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 2416
I20260810 23:04:07.012462 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:04:07.012661 23777 external_mini_cluster.cc:1428] Setting key 74ac84797f950e56235a3cc2d4c39592
I20260810 23:04:07.012769 2344 ts_manager.cc:194] Registered new tserver with Master: e2923a6841ed4373bc38916756befb93 (127.23.56.65:37263)
I20260810 23:04:07.013907 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34473
--builtin_ntp_servers=127.23.56.84:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:04:07.096598 2568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:07.096796 2568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:07.096825 2568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:07.097982 2568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:07.098021 2568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:07.098047 2568 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:04:07.099586 2568 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:07.099829 2568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:07.100128 2568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:07.105136 2573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:07.105207 2574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:07.105144 2576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:07.105440 2568 server_base.cc:1034] running on GCE node
I20260810 23:04:07.105631 2568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:07.105870 2568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:07.107076 2568 hybrid_clock.cc:648] HybridClock initialized: now 1786403047107071 us; error 53 us; skew 500 ppm
I20260810 23:04:07.108502 2568 webserver.cc:460] Webserver started at http://127.23.56.66:37379/ using document root <none> and password file <none>
I20260810 23:04:07.108744 2568 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:07.108821 2568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:07.108968 2568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:07.110093 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "c32a8c68499b4e2797c2e2491bbb9bdc"
format_stamp: "Formatted at 2026-08-10 23:04:07 on dist-test-slave-1xrh"
server_key: "08561be4021142c7b4e5b56cabe8fd1b"
server_key_iv: "1262cc7e3a133a7a397aa0565b8c9229"
server_key_version: "encryptionkey@0"
I20260810 23:04:07.110513 2568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "c32a8c68499b4e2797c2e2491bbb9bdc"
format_stamp: "Formatted at 2026-08-10 23:04:07 on dist-test-slave-1xrh"
server_key: "08561be4021142c7b4e5b56cabe8fd1b"
server_key_iv: "1262cc7e3a133a7a397aa0565b8c9229"
server_key_version: "encryptionkey@0"
I20260810 23:04:07.111922 2568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260810 23:04:07.112629 2583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:07.112804 2568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:04:07.112910 2568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "c32a8c68499b4e2797c2e2491bbb9bdc"
format_stamp: "Formatted at 2026-08-10 23:04:07 on dist-test-slave-1xrh"
server_key: "08561be4021142c7b4e5b56cabe8fd1b"
server_key_iv: "1262cc7e3a133a7a397aa0565b8c9229"
server_key_version: "encryptionkey@0"
I20260810 23:04:07.112984 2568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:07.116360 2568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:07.116621 2568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:07.116760 2568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:07.116971 2568 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:07.117264 2568 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:07.117324 2568 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:07.117364 2568 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:07.117391 2568 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:07.122973 2568 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:46489
I20260810 23:04:07.123083 2718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:46489 every 8 connection(s)
I20260810 23:04:07.123396 2568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:04:07.126685 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:07.124269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:58419 (local address 127.23.56.126:34473)
0810 23:04:07.124362 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:07.124364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:07.124533 (+ 169us) server_negotiation.cc:407] Connection header received
0810 23:04:07.125086 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:07.125087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:07.125093 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:07.125150 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:07.126387 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:07.126389 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:07.126389 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:04:07.126394 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:07.126417 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:04:07.126422 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:07.126501 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:07.126597 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 23:04:07.126613 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 23:04:07.126873 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:07.124162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34473 (local address 127.23.56.66:58419)
0810 23:04:07.124403 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:07.124447 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:07.124978 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:07.125225 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:07.125229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:07.125237 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:07.126230 (+ 993us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:04:07.126230 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:07.126257 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:04:07.126283 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:07.126498 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:07.126506 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:07.126509 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:04:07.126606 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 23:04:07.126666 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260810 23:04:07.127421 2719 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34473
I20260810 23:04:07.127503 2719 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:07.127684 2719 heartbeater.cc:511] Master 127.23.56.126:34473 requested a full tablet report, sending...
I20260810 23:04:07.128058 2343 ts_manager.cc:194] Registered new tserver with Master: c32a8c68499b4e2797c2e2491bbb9bdc (127.23.56.66:46489)
I20260810 23:04:07.129299 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 2568
I20260810 23:04:07.129423 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:04:07.129619 23777 external_mini_cluster.cc:1428] Setting key 227c31ce283b68ed9ecf9f4681c2d731
I20260810 23:04:07.130453 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34473
--builtin_ntp_servers=127.23.56.84:36461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 23:04:07.213052 2723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:07.213253 2723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:07.213274 2723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:07.214434 2723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:07.214475 2723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:07.214500 2723 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:04:07.216030 2723 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34473
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2723
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:07.216264 2723 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:07.216578 2723 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:07.221606 2732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:07.221580 2730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:07.221588 2729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:07.221694 2723 server_base.cc:1034] running on GCE node
I20260810 23:04:07.221977 2723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:07.222215 2723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:07.223371 2723 hybrid_clock.cc:648] HybridClock initialized: now 1786403047223337 us; error 47 us; skew 500 ppm
I20260810 23:04:07.225153 2723 webserver.cc:460] Webserver started at http://127.23.56.67:42893/ using document root <none> and password file <none>
I20260810 23:04:07.225472 2723 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:07.225544 2723 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:07.225685 2723 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:07.226729 2723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "d21d0768608e45cd9e19a466d9cb24bf"
format_stamp: "Formatted at 2026-08-10 23:04:07 on dist-test-slave-1xrh"
server_key: "714aff0fa3999de400828123bfc46eb0"
server_key_iv: "2e2b314365b8ff5e7fe2d252cb332639"
server_key_version: "encryptionkey@0"
I20260810 23:04:07.227113 2723 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "d21d0768608e45cd9e19a466d9cb24bf"
format_stamp: "Formatted at 2026-08-10 23:04:07 on dist-test-slave-1xrh"
server_key: "714aff0fa3999de400828123bfc46eb0"
server_key_iv: "2e2b314365b8ff5e7fe2d252cb332639"
server_key_version: "encryptionkey@0"
I20260810 23:04:07.228410 2723 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:04:07.229107 2738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:07.229287 2723 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:04:07.229383 2723 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "d21d0768608e45cd9e19a466d9cb24bf"
format_stamp: "Formatted at 2026-08-10 23:04:07 on dist-test-slave-1xrh"
server_key: "714aff0fa3999de400828123bfc46eb0"
server_key_iv: "2e2b314365b8ff5e7fe2d252cb332639"
server_key_version: "encryptionkey@0"
I20260810 23:04:07.229470 2723 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:07.232853 2723 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:07.233101 2723 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:07.233245 2723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:07.233451 2723 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:07.233819 2723 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:07.233873 2723 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:07.233913 2723 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:07.233942 2723 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:07.239827 2723 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:35463
I20260810 23:04:07.239879 2876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:35463 every 8 connection(s)
I20260810 23:04:07.240188 2723 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:04:07.243153 2413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:07.240941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:50305 (local address 127.23.56.126:34473)
0810 23:04:07.241024 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:07.241027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:07.241193 (+ 166us) server_negotiation.cc:407] Connection header received
0810 23:04:07.241614 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:07.241615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:07.241621 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:07.241670 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:07.242864 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:07.242865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:07.242865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:04:07.242870 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:07.242885 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:04:07.242901 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:07.242960 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:07.243035 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 23:04:07.243048 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 23:04:07.243391 2880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:07.240907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34473 (local address 127.23.56.67:50305)
0810 23:04:07.241082 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:07.241112 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:07.241522 (+ 410us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:07.241716 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:07.241718 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:07.241725 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:07.242646 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:04:07.242647 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:07.242672 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:04:07.242699 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:07.242979 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:07.242983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:07.242986 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:04:07.243042 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 23:04:07.243111 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":98,"threads_started":1}
I20260810 23:04:07.243820 2877 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34473
I20260810 23:04:07.243908 2877 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:07.244020 2877 heartbeater.cc:511] Master 127.23.56.126:34473 requested a full tablet report, sending...
I20260810 23:04:07.244372 2343 ts_manager.cc:194] Registered new tserver with Master: d21d0768608e45cd9e19a466d9cb24bf (127.23.56.67:35463)
I20260810 23:04:07.245793 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 2723
I20260810 23:04:07.245918 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:04:07.246116 23777 external_mini_cluster.cc:1428] Setting key 5b60d52589b3b7ce2aa8ab0995ee449a
I20260810 23:04:07.247023 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 23:04:07.249135 2413 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:04:07.248562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41582 (local address 127.23.56.126:34473)
0810 23:04:07.248631 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:07.248634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:07.248758 (+ 124us) server_negotiation.cc:407] Connection header received
0810 23:04:07.248852 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:07.248853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:07.248859 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:07.248921 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:07.249052 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41582: BlockingRecv error: recv got EOF from 127.0.0.1:41582 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 23:04:07.249164 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 2416
I20260810 23:04:07.259696 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 2568
I20260810 23:04:07.266474 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 2723
I20260810 23:04:07.272096 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 2306
2026-08-10T23:04:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (554 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:04:07 dist-test-slave-1xrh krb5kdc[2889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:04:07 dist-test-slave-1xrh krb5kdc[2889](info): set up 2 sockets
Aug 10 23:04:07 dist-test-slave-1xrh krb5kdc[2889](info): commencing operation
krb5kdc: starting...
W20260810 23:04:08.378973 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:04:08 dist-test-slave-1xrh krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:04:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:04:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:04:08.500170 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:43863
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:43863
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:08.584554 2902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:08.584748 2902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:08.584769 2902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:08.585815 2902 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:08.585850 2902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:08.585865 2902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:08.585877 2902 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:08.585889 2902 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:08.587325 2902 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:43863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:43863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:08.587507 2902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:08.587666 2902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:04:08.592510 2902 server_base.cc:1034] running on GCE node
W20260810 23:04:08.592475 2907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:08.592437 2908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:08.592460 2910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:08.593003 2902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:08.593284 2902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:08.594457 2902 hybrid_clock.cc:648] HybridClock initialized: now 1786403048594453 us; error 35 us; skew 500 ppm
Aug 10 23:04:08 dist-test-slave-1xrh krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:08.596956 2902 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:04:08.597402 2902 webserver.cc:460] Webserver started at http://127.23.56.126:38367/ using document root <none> and password file <none>
I20260810 23:04:08.597622 2902 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:08.597688 2902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:08.597837 2902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:08.598855 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "ac887fc7711f4427ad5d54a32ead4a99"
format_stamp: "Formatted at 2026-08-10 23:04:08 on dist-test-slave-1xrh"
server_key: "17201c6d923b58d3a0085a91b470bc14"
server_key_iv: "207f64731cf828f2490faf224aa59c0d"
server_key_version: "encryptionkey@0"
I20260810 23:04:08.599277 2902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "ac887fc7711f4427ad5d54a32ead4a99"
format_stamp: "Formatted at 2026-08-10 23:04:08 on dist-test-slave-1xrh"
server_key: "17201c6d923b58d3a0085a91b470bc14"
server_key_iv: "207f64731cf828f2490faf224aa59c0d"
server_key_version: "encryptionkey@0"
I20260810 23:04:08.600612 2902 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:04:08.601343 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:08.601531 2902 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:04:08.601627 2902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "ac887fc7711f4427ad5d54a32ead4a99"
format_stamp: "Formatted at 2026-08-10 23:04:08 on dist-test-slave-1xrh"
server_key: "17201c6d923b58d3a0085a91b470bc14"
server_key_iv: "207f64731cf828f2490faf224aa59c0d"
server_key_version: "encryptionkey@0"
I20260810 23:04:08.601711 2902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:08.655725 2902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:08.658149 2902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:08.658317 2902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:08.662513 2902 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:43863
I20260810 23:04:08.662567 2981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:43863 every 8 connection(s)
I20260810 23:04:08.662935 2902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:08.664098 2982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:08.665414 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 2902
I20260810 23:04:08.665522 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:04:08.665745 23777 external_mini_cluster.cc:1428] Setting key 3d0a3647b81172f98a2270bb9e5a963e
I20260810 23:04:08.667284 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99: Bootstrap starting.
I20260810 23:04:08.668113 2982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:08.668444 2982 log.cc:824] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:08.669251 2982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99: No bootstrap required, opened a new log
I20260810 23:04:08.671984 2982 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887fc7711f4427ad5d54a32ead4a99" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43863 } }
I20260810 23:04:08.672178 2982 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:08.672217 2982 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac887fc7711f4427ad5d54a32ead4a99, State: Initialized, Role: FOLLOWER
I20260810 23:04:08.672375 2982 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887fc7711f4427ad5d54a32ead4a99" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43863 } }
I20260810 23:04:08.672498 2982 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:08.672550 2982 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:08.672606 2982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [term 0 FOLLOWER]: Advancing to term 1
Aug 10 23:04:08 dist-test-slave-1xrh krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403048, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:04:08.673242 2982 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887fc7711f4427ad5d54a32ead4a99" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43863 } }
I20260810 23:04:08.673378 2982 leader_election.cc:302] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac887fc7711f4427ad5d54a32ead4a99; no voters:
I20260810 23:04:08.673583 2982 leader_election.cc:288] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:04:08.673645 2992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:08.673825 2992 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [term 1 LEADER]: Becoming Leader. State: Replica: ac887fc7711f4427ad5d54a32ead4a99, State: Running, Role: LEADER
I20260810 23:04:08.673936 2992 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887fc7711f4427ad5d54a32ead4a99" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43863 } }
I20260810 23:04:08.674029 2982 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:08.674239 2992 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac887fc7711f4427ad5d54a32ead4a99. Latest consensus state: current_term: 1 leader_uuid: "ac887fc7711f4427ad5d54a32ead4a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887fc7711f4427ad5d54a32ead4a99" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43863 } } }
I20260810 23:04:08.674248 2995 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac887fc7711f4427ad5d54a32ead4a99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac887fc7711f4427ad5d54a32ead4a99" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43863 } } }
I20260810 23:04:08.674433 2995 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:08.674662 2992 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99 [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:08.674850 2997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:08.675632 2997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:08.677315 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:08.666298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48484 (local address 127.23.56.126:43863)
0810 23:04:08.666491 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:08.666499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:08.666523 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:04:08.666934 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:08.666939 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:08.666983 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:08.667096 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:08.668015 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:08.669002 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:08.670125 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:08.670523 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:08.670602 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:08.673467 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:08.673471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:08.673472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:08.673486 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:08.675798 (+ 2312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:08.676164 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:08.676168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:08.676168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:08.676284 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:08.676491 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:08.676492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:08.676492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:08.676723 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:08.676867 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:08.676993 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 23:04:08.677131 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260810 23:04:08.677368 2997 catalog_manager.cc:1347] Generated new cluster ID: e5ee48f050f84ff6938808c7498cb249
I20260810 23:04:08.677413 2997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:04:08.714468 2997 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:04:08.715260 2997 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:04:08.726146 2997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99: Generated new TSK 0
I20260810 23:04:08.726344 2997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:04:08.761905 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43863
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:08.851263 3014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:08.851477 3014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:08.851513 3014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:08.852658 3014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:08.852721 3014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:08.852767 3014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:04:08.854249 3014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:08.854516 3014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:08.854746 3014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:08.859751 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:08.859769 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:08.859743 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:08.859953 3014 server_base.cc:1034] running on GCE node
I20260810 23:04:08.860181 3014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:08.860426 3014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:08.861590 3014 hybrid_clock.cc:648] HybridClock initialized: now 1786403048861598 us; error 48 us; skew 500 ppm
Aug 10 23:04:08 dist-test-slave-1xrh krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:08.864171 3014 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:04:08.864579 3014 webserver.cc:460] Webserver started at http://127.23.56.65:37041/ using document root <none> and password file <none>
I20260810 23:04:08.864818 3014 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:08.864886 3014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:08.865047 3014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:08.866034 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "32c456746c384c0583409ae0bc6f892a"
format_stamp: "Formatted at 2026-08-10 23:04:08 on dist-test-slave-1xrh"
server_key: "0aaad2e766e4d6e05d8072bf93a82100"
server_key_iv: "19d28b327cf9bb160557b6a92823e9ff"
server_key_version: "encryptionkey@0"
I20260810 23:04:08.866387 3014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "32c456746c384c0583409ae0bc6f892a"
format_stamp: "Formatted at 2026-08-10 23:04:08 on dist-test-slave-1xrh"
server_key: "0aaad2e766e4d6e05d8072bf93a82100"
server_key_iv: "19d28b327cf9bb160557b6a92823e9ff"
server_key_version: "encryptionkey@0"
I20260810 23:04:08.867789 3014 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:04:08.868569 3032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:08.868767 3014 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:04:08.868863 3014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "32c456746c384c0583409ae0bc6f892a"
format_stamp: "Formatted at 2026-08-10 23:04:08 on dist-test-slave-1xrh"
server_key: "0aaad2e766e4d6e05d8072bf93a82100"
server_key_iv: "19d28b327cf9bb160557b6a92823e9ff"
server_key_version: "encryptionkey@0"
I20260810 23:04:08.868939 3014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:08.898097 3014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:08.900671 3014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:08.900826 3014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:08.901146 3014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:08.901538 3014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:08.901587 3014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:08.901629 3014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:08.901652 3014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:08.908533 3014 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:38027
I20260810 23:04:08.908581 3163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:38027 every 8 connection(s)
I20260810 23:04:08.908897 3014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
Aug 10 23:04:08 dist-test-slave-1xrh krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403048, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:04:08.917747 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 3014
I20260810 23:04:08.917840 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:04:08.918047 23777 external_mini_cluster.cc:1428] Setting key 2080f8cd4ccefcca77aa5895b9820b2a
I20260810 23:04:08.918419 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:08.909688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:46269 (local address 127.23.56.126:43863)
0810 23:04:08.909747 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:08.909749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:08.909991 (+ 242us) server_negotiation.cc:407] Connection header received
0810 23:04:08.910558 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:08.910559 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:08.910573 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:08.910634 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:08.911641 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:08.912160 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:08.913186 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:08.913416 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:08.913496 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:08.915560 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:08.915562 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:08.915563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:08.915571 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:08.916916 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:08.917305 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:08.917306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:08.917306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:08.917355 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:08.917716 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:08.917716 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:08.917717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:08.917837 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:08.917941 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:08.918244 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 23:04:08.918328 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 23:04:08.918618 3167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:08.909661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:43863 (local address 127.23.56.65:46269)
0810 23:04:08.909868 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:08.909906 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:08.910442 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:08.910706 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:08.910709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:08.911340 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:08.911537 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:08.911543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:08.912276 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:08.912278 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:08.913076 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:08.913078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:08.913505 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:08.913506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:08.913630 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:08.914061 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:08.914070 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:08.915467 (+ 1397us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:08.917001 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:08.917008 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:08.917009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:08.917223 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:08.917442 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:08.917443 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:08.917443 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:08.917638 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:08.917958 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:08.917965 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:08.918089 (+ 124us) client_negotiation.cc:769] Sending connection context
0810 23:04:08.918207 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:04:08.918383 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":80,"threads_started":1}
I20260810 23:04:08.919195 3165 heartbeater.cc:347] Connected to a master server at 127.23.56.126:43863
I20260810 23:04:08.919315 3165 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:08.919502 3165 heartbeater.cc:511] Master 127.23.56.126:43863 requested a full tablet report, sending...
I20260810 23:04:08.920033 2938 ts_manager.cc:194] Registered new tserver with Master: 32c456746c384c0583409ae0bc6f892a (127.23.56.65:38027)
I20260810 23:04:08.920887 2938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:46269
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:04:08.945469 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43863
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:09.026887 3170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:09.027101 3170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:09.027179 3170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:09.028340 3170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:09.028383 3170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:09.028436 3170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:04:09.029908 3170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:09.030153 3170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:09.030382 3170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:09.035157 3175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:09.035219 3170 server_base.cc:1034] running on GCE node
W20260810 23:04:09.035151 3178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:09.035166 3176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:09.035562 3170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:09.035754 3170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:09.036903 3170 hybrid_clock.cc:648] HybridClock initialized: now 1786403049036889 us; error 31 us; skew 500 ppm
Aug 10 23:04:09 dist-test-slave-1xrh krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403049, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:09.039630 3170 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:04:09.040141 3170 webserver.cc:460] Webserver started at http://127.23.56.66:35257/ using document root <none> and password file <none>
I20260810 23:04:09.040370 3170 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:09.040436 3170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:09.040589 3170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:09.041625 3170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "78e00498eed542e9b5e7d882002ac3e0"
format_stamp: "Formatted at 2026-08-10 23:04:09 on dist-test-slave-1xrh"
server_key: "b8d8bdba0de876c49761735ad631fef1"
server_key_iv: "b9d810a28a514020fecf10bde6d23805"
server_key_version: "encryptionkey@0"
I20260810 23:04:09.042018 3170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "78e00498eed542e9b5e7d882002ac3e0"
format_stamp: "Formatted at 2026-08-10 23:04:09 on dist-test-slave-1xrh"
server_key: "b8d8bdba0de876c49761735ad631fef1"
server_key_iv: "b9d810a28a514020fecf10bde6d23805"
server_key_version: "encryptionkey@0"
I20260810 23:04:09.043397 3170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:04:09.044098 3186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:09.044265 3170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:04:09.044369 3170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "78e00498eed542e9b5e7d882002ac3e0"
format_stamp: "Formatted at 2026-08-10 23:04:09 on dist-test-slave-1xrh"
server_key: "b8d8bdba0de876c49761735ad631fef1"
server_key_iv: "b9d810a28a514020fecf10bde6d23805"
server_key_version: "encryptionkey@0"
I20260810 23:04:09.044447 3170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:09.067939 3170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:09.070304 3170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:09.070469 3170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:09.070712 3170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:09.071043 3170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:09.071106 3170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:09.071167 3170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:09.071197 3170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:09.077622 3170 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:45041
I20260810 23:04:09.077674 3318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:45041 every 8 connection(s)
I20260810 23:04:09.077991 3170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:04:09.080770 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 3170
I20260810 23:04:09.080861 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:04:09.081032 23777 external_mini_cluster.cc:1428] Setting key 92f2979027c25ceebd4b5970fc1bd4db
Aug 10 23:04:09 dist-test-slave-1xrh krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403049, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:04:09.088984 3321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.078634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:43863 (local address 127.23.56.66:48517)
0810 23:04:09.078782 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.078829 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:09.079365 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:09.079616 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:09.079621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:09.080194 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:09.080378 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.080383 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.081363 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.081365 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.082484 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.082488 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.082984 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.082985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.083101 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.083801 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:09.083813 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:09.085528 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:09.087566 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:09.087573 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:09.087575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:09.087811 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:09.088050 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:09.088051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:09.088051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:09.088154 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:09.088456 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:09.088462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:09.088582 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 23:04:09.088670 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:04:09.088797 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260810 23:04:09.088967 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.078676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:48517 (local address 127.23.56.126:43863)
0810 23:04:09.078727 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:09.078729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:09.078920 (+ 191us) server_negotiation.cc:407] Connection header received
0810 23:04:09.079475 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:09.079476 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:09.079490 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:09.079543 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:09.080483 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.081256 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.082585 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.082892 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.082968 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.085630 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:09.085632 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:09.085632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:09.085641 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:09.087474 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:09.087897 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:09.087899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:09.087899 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:09.087968 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:09.088222 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:09.088223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:09.088224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:09.088382 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:09.088469 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:09.088720 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 23:04:09.088827 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:04:09.089578 3319 heartbeater.cc:347] Connected to a master server at 127.23.56.126:43863
I20260810 23:04:09.089689 3319 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:09.089903 3319 heartbeater.cc:511] Master 127.23.56.126:43863 requested a full tablet report, sending...
I20260810 23:04:09.090451 2938 ts_manager.cc:194] Registered new tserver with Master: 78e00498eed542e9b5e7d882002ac3e0 (127.23.56.66:45041)
I20260810 23:04:09.090842 2938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:48517
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:04:09.110471 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43863
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:09.191820 3324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:09.192013 3324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:09.192036 3324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:09.193102 3324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:09.193135 3324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:09.193159 3324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:04:09.194649 3324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43863
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3324
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:09.194828 3324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:09.195060 3324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:09.199954 3331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:09.200094 3334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:09.200048 3332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:09.200403 3324 server_base.cc:1034] running on GCE node
I20260810 23:04:09.200645 3324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:09.200877 3324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:09.202106 3324 hybrid_clock.cc:648] HybridClock initialized: now 1786403049202093 us; error 42 us; skew 500 ppm
Aug 10 23:04:09 dist-test-slave-1xrh krb5kdc[2889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403049, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:09.204679 3324 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:04:09.205111 3324 webserver.cc:460] Webserver started at http://127.23.56.67:45841/ using document root <none> and password file <none>
I20260810 23:04:09.205341 3324 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:09.205423 3324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:09.205578 3324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:04:09.206586 3324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "65f5b7a280a4491bb3c2b9186266dea4"
format_stamp: "Formatted at 2026-08-10 23:04:09 on dist-test-slave-1xrh"
server_key: "a09b06388d88201ec7250029e1860447"
server_key_iv: "5533626b6ffcc6af9dc7afe2724be88e"
server_key_version: "encryptionkey@0"
I20260810 23:04:09.206952 3324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "65f5b7a280a4491bb3c2b9186266dea4"
format_stamp: "Formatted at 2026-08-10 23:04:09 on dist-test-slave-1xrh"
server_key: "a09b06388d88201ec7250029e1860447"
server_key_iv: "5533626b6ffcc6af9dc7afe2724be88e"
server_key_version: "encryptionkey@0"
I20260810 23:04:09.208243 3324 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:04:09.208915 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:09.209082 3324 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:04:09.209182 3324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "65f5b7a280a4491bb3c2b9186266dea4"
format_stamp: "Formatted at 2026-08-10 23:04:09 on dist-test-slave-1xrh"
server_key: "a09b06388d88201ec7250029e1860447"
server_key_iv: "5533626b6ffcc6af9dc7afe2724be88e"
server_key_version: "encryptionkey@0"
I20260810 23:04:09.209251 3324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:09.221331 3324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:09.223007 3324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:09.223201 3324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:09.223419 3324 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:09.223712 3324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:09.223764 3324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:09.223825 3324 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:09.223855 3324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:09.229853 3324 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:45813
I20260810 23:04:09.229889 3468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:45813 every 8 connection(s)
I20260810 23:04:09.230202 3324 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:04:09.235558 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 3324
I20260810 23:04:09.235662 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:04:09.235827 23777 external_mini_cluster.cc:1428] Setting key 8ab12c12a7a20a34ed0f2a03cbac2e6d
Aug 10 23:04:09 dist-test-slave-1xrh krb5kdc[2889](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403049, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:04:09.239362 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.230959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:36729 (local address 127.23.56.126:43863)
0810 23:04:09.231020 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:09.231022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:09.231208 (+ 186us) server_negotiation.cc:407] Connection header received
0810 23:04:09.231695 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:09.231696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:09.231710 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:09.231790 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:09.232701 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.233190 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.234116 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.234291 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.234339 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.236765 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:09.236768 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:09.236768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:09.236777 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:09.238010 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:09.238391 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:09.238392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:09.238393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:09.238451 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:09.238701 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:09.238702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:09.238702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:09.238815 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:09.238891 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:09.239188 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 23:04:09.239273 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:04:09.239521 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.230932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:43863 (local address 127.23.56.67:36729)
0810 23:04:09.231081 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.231141 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:09.231598 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:09.231891 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:09.231894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:09.232437 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:09.232621 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.232625 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.233280 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.233281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.234044 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.234046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.234366 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.234367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.234461 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.234887 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:09.234895 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:09.236683 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:09.238092 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:09.238099 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:09.238101 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:09.238314 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:09.238522 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:09.238523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:09.238523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:09.238634 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:09.238882 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:09.238889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:09.239026 (+ 137us) client_negotiation.cc:769] Sending connection context
0810 23:04:09.239127 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:04:09.239293 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260810 23:04:09.240000 3469 heartbeater.cc:347] Connected to a master server at 127.23.56.126:43863
I20260810 23:04:09.240077 3469 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:09.240268 3469 heartbeater.cc:511] Master 127.23.56.126:43863 requested a full tablet report, sending...
I20260810 23:04:09.240621 2938 ts_manager.cc:194] Registered new tserver with Master: 65f5b7a280a4491bb3c2b9186266dea4 (127.23.56.67:45813)
I20260810 23:04:09.241024 2938 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:36729
I20260810 23:04:09.247535 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:04:09.254437 2988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.248893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.23.56.126:43863)
0810 23:04:09.248941 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:09.248944 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:09.249030 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:04:09.249079 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:09.249080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:09.249088 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:09.249133 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:09.249632 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.250131 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.250821 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.250991 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.251039 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.252227 (+ 1188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:09.252228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:09.252229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:09.252234 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:09.253416 (+ 1182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:09.253724 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:09.253724 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:09.253725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:09.253772 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:09.253996 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:09.253997 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:09.253997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:09.254129 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:09.254205 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:09.254292 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:04:09.254350 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:04:09.255930 2938 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:04:09.257496 2938 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:04:09.264164 3484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.259940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:38027 (local address 127.0.0.1:41510)
0810 23:04:09.260377 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.260386 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:09.260432 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:09.260722 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:09.260725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:09.260727 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:09.260799 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.260804 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.261754 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.261755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.262962 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.262964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.263838 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.263839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.263920 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.263934 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 23:04:09.264037 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:04:09.264091 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":7,"thread_start_us":71,"threads_started":1}
I20260810 23:04:09.264173 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.260118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.23.56.65:38027)
0810 23:04:09.260334 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:09.260340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:09.260499 (+ 159us) server_negotiation.cc:407] Connection header received
0810 23:04:09.260545 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:09.260548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:09.260564 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:09.260635 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:09.260892 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.261656 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.263054 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.263706 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.263812 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.263875 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:09.264056 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 23:04:09.264096 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
I20260810 23:04:09.264457 3488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.260146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:45041 (local address 127.0.0.1:50498)
0810 23:04:09.260516 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.260524 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:09.260571 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:09.260746 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:09.260747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:09.260748 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:09.260799 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.260801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.262098 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.262099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.263190 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.263192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.264202 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.264204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.264277 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.264283 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:04:09.264339 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 23:04:09.264390 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":6,"thread_start_us":75,"threads_started":1}
I20260810 23:04:09.264719 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.259940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:45813 (local address 127.0.0.1:43508)
0810 23:04:09.260285 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.260326 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:09.260411 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:09.261035 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:09.261036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:09.261038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:09.261074 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.261075 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.262263 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.262264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.263288 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.263292 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.264206 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.264206 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.264565 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.264571 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 23:04:09.264623 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 23:04:09.264657 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":288,"thread_start_us":59,"threads_started":1}
I20260810 23:04:09.264736 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.260268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50498 (local address 127.23.56.66:45041)
0810 23:04:09.260478 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:09.260484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:09.260570 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:04:09.260639 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:09.260641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:09.260657 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:09.260739 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:09.260869 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.261982 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.263262 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.264032 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.264108 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.264164 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:09.264614 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 23:04:09.264661 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":129,"threads_started":1}
I20260810 23:04:09.265007 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.260087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.23.56.67:45813)
0810 23:04:09.260520 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:09.260524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:09.260533 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:04:09.260571 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:09.260573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:09.260590 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:09.261022 (+ 432us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:09.261139 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.262154 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.263373 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.264074 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.264132 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.264177 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:09.264888 (+ 711us) server_negotiation.cc:299] Negotiation successful
0810 23:04:09.264929 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":128,"threads_started":1}
I20260810 23:04:09.264952 3089 tablet_service.cc:1467] Processing CreateTablet for tablet f55343effd52450b80fa2d042894a7fc (DEFAULT_TABLE table=test-table [id=0057131378944240a1e376771c8af282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:09.265331 3089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55343effd52450b80fa2d042894a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:09.265339 3243 tablet_service.cc:1467] Processing CreateTablet for tablet f55343effd52450b80fa2d042894a7fc (DEFAULT_TABLE table=test-table [id=0057131378944240a1e376771c8af282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:09.265542 3385 tablet_service.cc:1467] Processing CreateTablet for tablet f55343effd52450b80fa2d042894a7fc (DEFAULT_TABLE table=test-table [id=0057131378944240a1e376771c8af282]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:09.265676 3243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55343effd52450b80fa2d042894a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:09.265830 3385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f55343effd52450b80fa2d042894a7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:09.268085 3491 tablet_bootstrap.cc:492] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a: Bootstrap starting.
I20260810 23:04:09.268901 3493 tablet_bootstrap.cc:492] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: Bootstrap starting.
I20260810 23:04:09.269044 3491 tablet_bootstrap.cc:654] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:09.269621 3493 tablet_bootstrap.cc:654] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:09.269634 3491 log.cc:824] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:09.270198 3493 log.cc:824] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:09.270439 3491 tablet_bootstrap.cc:492] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a: No bootstrap required, opened a new log
I20260810 23:04:09.270515 3491 ts_tablet_manager.cc:1397] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:04:09.270527 3492 tablet_bootstrap.cc:492] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: Bootstrap starting.
I20260810 23:04:09.270941 3493 tablet_bootstrap.cc:492] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: No bootstrap required, opened a new log
I20260810 23:04:09.271013 3493 ts_tablet_manager.cc:1397] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:04:09.271175 3492 tablet_bootstrap.cc:654] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:09.271764 3492 log.cc:824] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:09.272598 3492 tablet_bootstrap.cc:492] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: No bootstrap required, opened a new log
I20260810 23:04:09.272670 3492 ts_tablet_manager.cc:1397] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:04:09.272866 3491 raft_consensus.cc:348] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } }
I20260810 23:04:09.272923 3493 raft_consensus.cc:348] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } }
I20260810 23:04:09.273041 3491 raft_consensus.cc:374] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:09.273061 3493 raft_consensus.cc:374] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:09.273077 3491 raft_consensus.cc:729] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c456746c384c0583409ae0bc6f892a, State: Initialized, Role: FOLLOWER
I20260810 23:04:09.273102 3493 raft_consensus.cc:729] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f5b7a280a4491bb3c2b9186266dea4, State: Initialized, Role: FOLLOWER
I20260810 23:04:09.273218 3493 consensus_queue.cc:260] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } }
I20260810 23:04:09.273231 3491 consensus_queue.cc:260] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } }
I20260810 23:04:09.273521 3491 ts_tablet_manager.cc:1428] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:04:09.273523 3493 ts_tablet_manager.cc:1428] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:04:09.273542 3165 heartbeater.cc:503] Master 127.23.56.126:43863 was elected leader, sending a full tablet report...
I20260810 23:04:09.273655 3469 heartbeater.cc:503] Master 127.23.56.126:43863 was elected leader, sending a full tablet report...
I20260810 23:04:09.274976 3492 raft_consensus.cc:348] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } }
I20260810 23:04:09.275158 3492 raft_consensus.cc:374] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:09.275218 3492 raft_consensus.cc:729] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78e00498eed542e9b5e7d882002ac3e0, State: Initialized, Role: FOLLOWER
I20260810 23:04:09.275331 3492 consensus_queue.cc:260] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } }
I20260810 23:04:09.275602 3319 heartbeater.cc:503] Master 127.23.56.126:43863 was elected leader, sending a full tablet report...
I20260810 23:04:09.275602 3492 ts_tablet_manager.cc:1428] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260810 23:04:09.328554 3320 tablet.cc:2369] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:04:09.409629 3166 tablet.cc:2369] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:04:09.463083 3497 raft_consensus.cc:482] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:09.463264 3497 raft_consensus.cc:504] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } }
I20260810 23:04:09.463708 3497 leader_election.cc:288] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32c456746c384c0583409ae0bc6f892a (127.23.56.65:38027), 78e00498eed542e9b5e7d882002ac3e0 (127.23.56.66:45041)
I20260810 23:04:09.466737 3489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.464009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:56103 (local address 127.23.56.65:38027)
0810 23:04:09.464066 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:09.464068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:09.464080 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:04:09.464125 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:09.464125 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:09.464138 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:09.464194 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:09.464413 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.464930 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.465975 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.466346 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.466401 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.466408 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:09.466585 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 23:04:09.466631 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:04:09.466753 3472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.463910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:38027 (local address 127.23.56.67:56103)
0810 23:04:09.463995 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.464008 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:09.464058 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:09.464233 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:09.464235 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:09.464237 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:09.464316 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.464327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.465007 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.465008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.465872 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.465875 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.466422 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.466423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.466507 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.466511 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:04:09.466566 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 23:04:09.466647 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":54}
I20260810 23:04:09.466885 3490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.464065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:48577 (local address 127.23.56.66:45041)
0810 23:04:09.464114 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:09.464117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:09.464272 (+ 155us) server_negotiation.cc:407] Connection header received
0810 23:04:09.464306 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:09.464307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:09.464327 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:09.464384 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:09.464547 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.465059 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.466051 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.466474 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.466550 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.466559 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:09.466748 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 23:04:09.466796 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:04:09.466907 3501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.463997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:45041 (local address 127.23.56.67:48577)
0810 23:04:09.464168 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.464178 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:09.464226 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:09.464403 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:09.464404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:09.464406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:09.464471 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.464473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.465176 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.465177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.465949 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:09.465951 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.466579 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.466579 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:09.466655 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.466660 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:04:09.466719 (+ 59us) client_negotiation.cc:240] Negotiation successful
0810 23:04:09.466760 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":102,"threads_started":1}
I20260810 23:04:09.467180 3112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55343effd52450b80fa2d042894a7fc" candidate_uuid: "65f5b7a280a4491bb3c2b9186266dea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c456746c384c0583409ae0bc6f892a" is_pre_election: true
I20260810 23:04:09.467360 3112 raft_consensus.cc:2454] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65f5b7a280a4491bb3c2b9186266dea4 in term 0.
I20260810 23:04:09.467336 3266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55343effd52450b80fa2d042894a7fc" candidate_uuid: "65f5b7a280a4491bb3c2b9186266dea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e00498eed542e9b5e7d882002ac3e0" is_pre_election: true
I20260810 23:04:09.467478 3266 raft_consensus.cc:2454] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 65f5b7a280a4491bb3c2b9186266dea4 in term 0.
I20260810 23:04:09.467609 3342 leader_election.cc:302] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c456746c384c0583409ae0bc6f892a, 65f5b7a280a4491bb3c2b9186266dea4; no voters:
I20260810 23:04:09.467779 3497 raft_consensus.cc:2781] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:04:09.467825 3497 raft_consensus.cc:482] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:04:09.467841 3497 raft_consensus.cc:3037] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:09.468519 3497 raft_consensus.cc:504] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } }
I20260810 23:04:09.468653 3497 leader_election.cc:288] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [CANDIDATE]: Term 1 election: Requested vote from peers 32c456746c384c0583409ae0bc6f892a (127.23.56.65:38027), 78e00498eed542e9b5e7d882002ac3e0 (127.23.56.66:45041)
I20260810 23:04:09.468816 3266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55343effd52450b80fa2d042894a7fc" candidate_uuid: "65f5b7a280a4491bb3c2b9186266dea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78e00498eed542e9b5e7d882002ac3e0"
I20260810 23:04:09.468868 3112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f55343effd52450b80fa2d042894a7fc" candidate_uuid: "65f5b7a280a4491bb3c2b9186266dea4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32c456746c384c0583409ae0bc6f892a"
I20260810 23:04:09.468927 3112 raft_consensus.cc:3037] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:09.468931 3266 raft_consensus.cc:3037] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:09.469699 3112 raft_consensus.cc:2454] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65f5b7a280a4491bb3c2b9186266dea4 in term 1.
I20260810 23:04:09.469759 3266 raft_consensus.cc:2454] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 65f5b7a280a4491bb3c2b9186266dea4 in term 1.
I20260810 23:04:09.469863 3342 leader_election.cc:302] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32c456746c384c0583409ae0bc6f892a, 65f5b7a280a4491bb3c2b9186266dea4; no voters:
I20260810 23:04:09.469954 3497 raft_consensus.cc:2781] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:09.470100 3497 raft_consensus.cc:686] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 1 LEADER]: Becoming Leader. State: Replica: 65f5b7a280a4491bb3c2b9186266dea4, State: Running, Role: LEADER
I20260810 23:04:09.470212 3497 consensus_queue.cc:237] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } }
I20260810 23:04:09.471256 2938 catalog_manager.cc:5495] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 65f5b7a280a4491bb3c2b9186266dea4 (127.23.56.67). New cstate: current_term: 1 leader_uuid: "65f5b7a280a4491bb3c2b9186266dea4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "65f5b7a280a4491bb3c2b9186266dea4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 45813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 } health_report { overall_health: UNKNOWN } } }
W20260810 23:04:09.480759 3471 tablet.cc:2369] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:04:09.489462 3487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.487190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54556 (local address 127.23.56.67:45813)
0810 23:04:09.487257 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:09.487259 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:09.487267 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:04:09.487297 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:09.487297 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:09.487314 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:09.487373 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:04:09.487515 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.488023 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.488743 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:09.488911 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:09.488958 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:09.489161 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:04:09.489252 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:04:09.489311 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:09.489380 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 23:04:09.489396 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:04:09.491240 3112 raft_consensus.cc:1261] T f55343effd52450b80fa2d042894a7fc P 32c456746c384c0583409ae0bc6f892a [term 1 FOLLOWER]: Refusing update from remote peer 65f5b7a280a4491bb3c2b9186266dea4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:09.491246 3266 raft_consensus.cc:1261] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [term 1 FOLLOWER]: Refusing update from remote peer 65f5b7a280a4491bb3c2b9186266dea4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:09.491607 3502 consensus_queue.cc:1035] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32c456746c384c0583409ae0bc6f892a" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:09.491750 3497 consensus_queue.cc:1035] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78e00498eed542e9b5e7d882002ac3e0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 45041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:09.494948 3507 mvcc.cc:205] Tried to move back new op lower bound from 7317106890713362432 to 7317106890630701056. Current Snapshot: MvccSnapshot[applied={T|T < 7317106890713362432 or (T in {7317106890713362432})}]
I20260810 23:04:09.496398 2937 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:09.496495 2937 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48492:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:09.497259 2937 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ac887fc7711f4427ad5d54a32ead4a99: Sending DeleteTablet for 3 replicas of tablet f55343effd52450b80fa2d042894a7fc
I20260810 23:04:09.497551 3385 tablet_service.cc:1514] Processing DeleteTablet for tablet f55343effd52450b80fa2d042894a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:09 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:43508
I20260810 23:04:09.497599 3089 tablet_service.cc:1514] Processing DeleteTablet for tablet f55343effd52450b80fa2d042894a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:09 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:41510
I20260810 23:04:09.497634 3243 tablet_service.cc:1514] Processing DeleteTablet for tablet f55343effd52450b80fa2d042894a7fc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:09 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:50498
I20260810 23:04:09.497844 3518 tablet_replica.cc:330] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: stopping tablet replica
I20260810 23:04:09.498112 3516 tablet_replica.cc:330] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: stopping tablet replica
I20260810 23:04:09.498118 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 3014
I20260810 23:04:09.498247 3516 raft_consensus.cc:2229] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:04:09.498346 3516 raft_consensus.cc:2258] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:09.501035 3516 ts_tablet_manager.cc:1905] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:09.503026 3516 ts_tablet_manager.cc:1918] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:09.503118 3516 log.cc:1196] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/f55343effd52450b80fa2d042894a7fc
I20260810 23:04:09.503415 3516 ts_tablet_manager.cc:1939] T f55343effd52450b80fa2d042894a7fc P 65f5b7a280a4491bb3c2b9186266dea4: Deleting consensus metadata
I20260810 23:04:09.503866 2923 catalog_manager.cc:4841] TS 65f5b7a280a4491bb3c2b9186266dea4 (127.23.56.67:45813): tablet f55343effd52450b80fa2d042894a7fc (table test-table [id=0057131378944240a1e376771c8af282]) successfully deleted
I20260810 23:04:09.505010 3518 raft_consensus.cc:2229] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:09.505146 3518 raft_consensus.cc:2258] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:09.505609 3518 ts_tablet_manager.cc:1905] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:09.507253 3518 ts_tablet_manager.cc:1918] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:09.507330 3518 log.cc:1196] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/f55343effd52450b80fa2d042894a7fc
I20260810 23:04:09.507642 3518 ts_tablet_manager.cc:1939] T f55343effd52450b80fa2d042894a7fc P 78e00498eed542e9b5e7d882002ac3e0: Deleting consensus metadata
I20260810 23:04:09.508065 2923 catalog_manager.cc:4841] TS 78e00498eed542e9b5e7d882002ac3e0 (127.23.56.66:45041): tablet f55343effd52450b80fa2d042894a7fc (table test-table [id=0057131378944240a1e376771c8af282]) successfully deleted
W20260810 23:04:09.511070 2921 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:38027 (error 108)
I20260810 23:04:09.511323 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 3170
I20260810 23:04:09.512318 3486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:09.511491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:38027 (local address 127.0.0.1:40116)
0810 23:04:09.512240 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:09.512279 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:38027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":729}
W20260810 23:04:09.512600 2921 catalog_manager.cc:4584] TS 32c456746c384c0583409ae0bc6f892a (127.23.56.65:38027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f55343effd52450b80fa2d042894a7fc: Network error: Client connection negotiation failed: client connection to 127.23.56.65:38027: connect: Connection refused (error 111)
I20260810 23:04:09.517900 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 3324
I20260810 23:04:09.525579 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 2902
2026-08-10T23:04:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2253 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:04:09 dist-test-slave-1xrh krb5kdc[3524](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:04:09 dist-test-slave-1xrh krb5kdc[3524](info): set up 2 sockets
Aug 10 23:04:09 dist-test-slave-1xrh krb5kdc[3524](info): commencing operation
krb5kdc: starting...
W20260810 23:04:10.625669 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:04:10 dist-test-slave-1xrh krb5kdc[3524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:04:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:04:10Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.56.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.56.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/rangeradmin_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/rangeradmin_127.23.56.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.56.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.56.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/rangerlookup_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/rangerlookup_127.23.56.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.56.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.56.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/HTTP_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/HTTP_127.23.56.84@KRBTEST.COM.keytab.
I20260810 23:04:10.793861 23777 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQ_MaNL/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/postgres -l logfile start
2026-08-10 23:04:13.415 UTC [3558] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:04:13.416 UTC [3558] LOG: listening on IPv4 address "127.23.56.84", port 59545
2026-08-10 23:04:13.420 UTC [3558] LOG: listening on Unix socket "/tmp/.s.PGSQL.59545"
2026-08-10 23:04:13.458 UTC [3564] LOG: database system was shut down at 2026-08-10 23:04:11 UTC
2026-08-10 23:04:13.463 UTC [3558] LOG: database system is ready to accept connections
I20260810 23:04:14.582209 23777 mini_postgres.cc:96] Postgres bound to 59545
2026-08-10 23:04:14.590 UTC [3574] FATAL: database "slave" does not exist
127.23.56.84:59545 - accepting connections
I20260810 23:04:14.591566 23777 mini_ranger.cc:161] Starting Ranger...
I20260810 23:04:14.607752 23777 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:04:15.090319 23777 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:04:15.090415 23777 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-admin
2026-08-10 23:04:15,394 [I] DB FLAVOR :POSTGRES
2026-08-10 23:04:15,395 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:04:15,395 [I] Checking connection..
2026-08-10 23:04:15,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:04:15,645 [I] Checking connection passed.
2026-08-10 23:04:15,645 [I] --------- Verifying version history table ---------
2026-08-10 23:04:15,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:04:15,905 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:04:15,905 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:04:16,135 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:04:16,136 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:04:16,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:04:16.393 UTC [3645] WARNING: there is no transaction in progress
2026-08-10 23:04:16,400 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:04:16,401 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:04:16,680 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:04:16,680 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:04:16,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:04:16,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:04:17,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1xrh.c.gcp-upstream.internal','N') ;"
2026-08-10 23:04:17,435 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:04:17,687 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:04:17,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:04:17,946 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:04:17,946 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:04:18,237 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:04:18,490 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:04:18,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:04:19.452 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.468 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.485 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.500 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.516 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.537 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.601 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.608 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.622 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.632 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.642 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.655 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.665 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.671 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.678 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:19.685 UTC [3768] WARNING: there is no transaction in progress
2026-08-10 23:04:20,293 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:04:20,294 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:04:20,548 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:04:20,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:04:20,800 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:04:20,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:04:21,050 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:04:21,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:04:21,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:04:21,552 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1xrh.c.gcp-upstream.internal';"
2026-08-10 23:04:21,791 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:04:21,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:04:22,052 [I] DB_PATCHES have already been applied
I20260810 23:04:22.058415 23777 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQ_MaNL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:04:22.058477 23777 mini_ranger.cc:193] Using host: 127.23.56.84
I20260810 23:04:22.062183 23777 mini_ranger.cc:227] Ranger admin URL: http://127.23.56.84:43495
Aug 10, 2026 11:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:04:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:04:23.230671 23777 mini_ranger.cc:160] Time spent starting Ranger: real 8.639s user 0.000s sys 0.011s
Aug 10, 2026 11:04:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:04:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 11:04:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:04:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:04:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:04:43.936502 23777 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.56.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.56.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/rangerkms_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/rangerkms_127.23.56.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.56.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.56.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.56.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/HTTP_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.56.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/HTTP_127.23.56.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:04:44 dist-test-slave-1xrh krb5kdc[3524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:04:44.025949 23777 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:04:44.036134 23777 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:04:44.969866 23777 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:04:44.969936 23777 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-kms
I20260810 23:04:44.969949 23777 mini_ranger_kms.cc:227] Using postgres at 127.23.56.84:59545
2026-08-10 23:04:45,499 [I] DB FLAVOR :POSTGRES
2026-08-10 23:04:45,500 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:04:45,500 [I] Checking connection
2026-08-10 23:04:45,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:04:45,747 [I] connection success
2026-08-10 23:04:45,747 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:04:45,748 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:04:45,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:04:46,010 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:04:46,010 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:04:46,010 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:04:46,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:59545/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:04:46,277 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:04:46.749342 23777 mini_ranger_kms.cc:310] Created kms service
I20260810 23:04:46.970435 23777 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:04:47.015053 23777 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:04:47 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:04:47.413362 23777 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:04:47.413484 23777 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:04:47.413508 23777 mini_ranger_kms.cc:240] Using host: 127.23.56.84
I20260810 23:04:47.418134 23777 mini_ranger_kms.cc:276] Ranger KMS PID: 3973
I20260810 23:04:47.418252 23777 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.56.84:38387
Aug 10, 2026 11:04:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:04:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:04:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:04:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:04:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:04:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:04:49 dist-test-slave-1xrh krb5kdc[3524](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.56.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:49.814615 23777 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.789s user 0.004s sys 0.249s
I20260810 23:04:49.814736 23777 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:04:49.814826 23777 mini_ranger_kms.cc:401] 127.23.56.84:38387/kms/v1/keys
Aug 10 23:04:50 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.56.84@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
Aug 10 23:04:50 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.56.84@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
Aug 10 23:04:53 dist-test-slave-1xrh krb5kdc[3524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:04:53.319416 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:43919
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:43919
--ranger_config_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:38387/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:53.465230 4028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:53.465639 4028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:53.465698 4028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:53.467638 4028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:04:53.467748 4028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:53.467818 4028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:53.467906 4028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:04:53.467953 4028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:04:53.470485 4028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.56.84:38387/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:43919
--ranger_config_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:43919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4028
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:53.470968 4028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:53.471400 4028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:53.480917 4035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:53.481031 4028 server_base.cc:1034] running on GCE node
W20260810 23:04:53.481228 4037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:53.480909 4034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:53.481896 4028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:53.482371 4028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:53.483582 4028 hybrid_clock.cc:648] HybridClock initialized: now 1786403093483560 us; error 51 us; skew 500 ppm
Aug 10 23:04:53 dist-test-slave-1xrh krb5kdc[3524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403093, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:53.492239 4028 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:04:53.493114 4028 webserver.cc:460] Webserver started at http://127.23.56.126:34423/ using document root <none> and password file <none>
I20260810 23:04:53.493465 4028 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:53.493532 4028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:53.493820 4028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:04:53 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403093, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:04:53.589572 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "f4734eeb31a249f199f917d1f66e63dc"
format_stamp: "Formatted at 2026-08-10 23:04:53 on dist-test-slave-1xrh"
server_key: "9ffa5a597da1de4cf47e9e320876414f"
server_key_iv: "b387f21d2d18ace72b1aad9bf09f5521"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:53.590458 4028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "f4734eeb31a249f199f917d1f66e63dc"
format_stamp: "Formatted at 2026-08-10 23:04:53 on dist-test-slave-1xrh"
server_key: "9ffa5a597da1de4cf47e9e320876414f"
server_key_iv: "b387f21d2d18ace72b1aad9bf09f5521"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:53.593954 4028 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.001s
I20260810 23:04:53.618566 4047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:53.619416 4028 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 23:04:53.619598 4028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "f4734eeb31a249f199f917d1f66e63dc"
format_stamp: "Formatted at 2026-08-10 23:04:53 on dist-test-slave-1xrh"
server_key: "9ffa5a597da1de4cf47e9e320876414f"
server_key_iv: "b387f21d2d18ace72b1aad9bf09f5521"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:53.619767 4028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:53.639421 4028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:53.642642 4028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:53.642918 4028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:53.651476 4028 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:43919
I20260810 23:04:53.651531 4105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:43919 every 8 connection(s)
I20260810 23:04:53.652100 4028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:04:53.653395 4106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:53.658870 4106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc: Bootstrap starting.
I20260810 23:04:53.660097 4106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:53.660857 4106 log.cc:824] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:53.661895 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 4028
I20260810 23:04:53.662093 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:04:53.662251 4106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc: No bootstrap required, opened a new log
I20260810 23:04:53.662359 23777 external_mini_cluster.cc:1428] Setting key b5d07073578bf466de54b418225c6b65
I20260810 23:04:53.667510 4106 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4734eeb31a249f199f917d1f66e63dc" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43919 } }
I20260810 23:04:53.667896 4106 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:53.667973 4106 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4734eeb31a249f199f917d1f66e63dc, State: Initialized, Role: FOLLOWER
I20260810 23:04:53.668255 4106 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4734eeb31a249f199f917d1f66e63dc" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43919 } }
I20260810 23:04:53.668393 4106 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:04:53.668465 4106 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:04:53.668550 4106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:53.669689 4106 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4734eeb31a249f199f917d1f66e63dc" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43919 } }
I20260810 23:04:53.669857 4106 leader_election.cc:302] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4734eeb31a249f199f917d1f66e63dc; no voters:
I20260810 23:04:53.670113 4106 leader_election.cc:288] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:04:53.670334 4112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:53.670512 4106 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:04:53.670588 4112 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [term 1 LEADER]: Becoming Leader. State: Replica: f4734eeb31a249f199f917d1f66e63dc, State: Running, Role: LEADER
I20260810 23:04:53.670760 4112 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4734eeb31a249f199f917d1f66e63dc" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43919 } }
I20260810 23:04:53.671554 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4734eeb31a249f199f917d1f66e63dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4734eeb31a249f199f917d1f66e63dc" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43919 } } }
I20260810 23:04:53.671684 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:53.671861 4112 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4734eeb31a249f199f917d1f66e63dc. Latest consensus state: current_term: 1 leader_uuid: "f4734eeb31a249f199f917d1f66e63dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4734eeb31a249f199f917d1f66e63dc" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 43919 } } }
I20260810 23:04:53.671945 4112 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc [sys.catalog]: This master's current role is: LEADER
I20260810 23:04:53.672123 4121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:04:53.672957 4121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:04:53.673712 4106 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
Aug 10 23:04:53 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:04:53.678587 4121 catalog_manager.cc:1347] Generated new cluster ID: 8fb5905da157411dae09a56a23f42a93
I20260810 23:04:53.678676 4121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:04:53.687990 4111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:53.663305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.23.56.126:43919)
0810 23:04:53.663631 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:53.663652 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:53.663715 (+ 63us) server_negotiation.cc:407] Connection header received
0810 23:04:53.664726 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:53.664734 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:53.664795 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:53.665070 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:53.666042 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:53.667363 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:53.668964 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:53.669600 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:53.670045 (+ 445us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:53.676090 (+ 6045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:53.676094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:53.676095 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:53.676139 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:53.683406 (+ 7267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:53.684246 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:53.684250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:53.684250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:53.684547 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:53.684882 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:53.684883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:53.684883 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:53.685673 (+ 790us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:53.686609 (+ 936us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:53.686718 (+ 109us) server_negotiation.cc:299] Negotiation successful
0810 23:04:53.687484 (+ 766us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":128,"threads_started":1}
I20260810 23:04:53.696980 4121 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:04:53.698211 4121 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:04:53.707566 4121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc: Generated new TSK 0
I20260810 23:04:53.707893 4121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:04:54.798120 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:38387/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43919
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:54.899509 4158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:54.899850 4158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:54.899923 4158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:54.901603 4158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:54.901659 4158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:54.901697 4158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:04:54.903831 4158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.56.84:38387/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:54.904287 4158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:54.904561 4158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:54.910959 4165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:54.911196 4166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:54.911279 4168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:54.912396 4158 server_base.cc:1034] running on GCE node
I20260810 23:04:54.912649 4158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:54.912985 4158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:54.914152 4158 hybrid_clock.cc:648] HybridClock initialized: now 1786403094914141 us; error 27 us; skew 500 ppm
Aug 10 23:04:54 dist-test-slave-1xrh krb5kdc[3524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403094, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:54.916772 4158 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:04:54.917276 4158 webserver.cc:460] Webserver started at http://127.23.56.65:36689/ using document root <none> and password file <none>
I20260810 23:04:54.917536 4158 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:54.917580 4158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:54.917709 4158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:04:54 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403094, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:04:54.939754 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "867d89767f304d9899b36f46739d613f"
format_stamp: "Formatted at 2026-08-10 23:04:54 on dist-test-slave-1xrh"
server_key: "095daef6c422cde0d28ea358ce9d4a8c"
server_key_iv: "9ed5525764daa1222367c1f5405f30fb"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:54.940260 4158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "867d89767f304d9899b36f46739d613f"
format_stamp: "Formatted at 2026-08-10 23:04:54 on dist-test-slave-1xrh"
server_key: "095daef6c422cde0d28ea358ce9d4a8c"
server_key_iv: "9ed5525764daa1222367c1f5405f30fb"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:54.942655 4158 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:04:54.953946 4177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:54.954145 4158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260810 23:04:54.954244 4158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "867d89767f304d9899b36f46739d613f"
format_stamp: "Formatted at 2026-08-10 23:04:54 on dist-test-slave-1xrh"
server_key: "095daef6c422cde0d28ea358ce9d4a8c"
server_key_iv: "9ed5525764daa1222367c1f5405f30fb"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:54.954341 4158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:54.987926 4158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:54.990451 4158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:54.990675 4158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:54.990978 4158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:54.991457 4158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:54.991549 4158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:54.991653 4158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:54.991732 4158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:54.998260 4158 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:42235
I20260810 23:04:54.998778 4158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:04:55.001737 4308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:42235 every 8 connection(s)
I20260810 23:04:55.005999 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 4158
I20260810 23:04:55.006099 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:04:55.006285 23777 external_mini_cluster.cc:1428] Setting key 237784dcee08e7caf8a48972e4b760a6
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Aug 10 23:04:55 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403094, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:04:55.033624 4311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.003818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:43919 (local address 127.23.56.65:45845)
0810 23:04:55.011337 (+ 7519us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:55.011397 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:55.012032 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:55.022622 (+ 10590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:55.022628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:55.022909 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:55.023240 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.023248 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.024374 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.024375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.025660 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.025664 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.026551 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.026553 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.026940 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.027710 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:55.027721 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:55.029679 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:55.031940 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:55.031948 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:55.031951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:55.032254 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:55.032491 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:55.032491 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:55.032492 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:55.032618 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:55.032945 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:55.032952 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:55.033115 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 23:04:55.033223 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:04:55.033382 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7353,"thread_start_us":91,"threads_started":1}
I20260810 23:04:55.034665 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.004138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:45845 (local address 127.23.56.126:43919)
0810 23:04:55.013364 (+ 9226us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.013366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.013377 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:04:55.013412 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.013413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.013425 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.013505 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:55.023379 (+ 9874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.024242 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.026113 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.026417 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.026534 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.029816 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:55.029817 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:55.029818 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:55.029827 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:55.031837 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:55.032352 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:55.032354 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:55.032354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:55.032422 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:55.032693 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:55.032694 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:55.032695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:55.032847 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:55.032914 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.034477 (+ 1563us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.034566 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9154,"thread_start_us":89,"threads_started":1}
I20260810 23:04:55.035068 4309 heartbeater.cc:347] Connected to a master server at 127.23.56.126:43919
I20260810 23:04:55.035197 4309 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:55.035414 4309 heartbeater.cc:511] Master 127.23.56.126:43919 requested a full tablet report, sending...
I20260810 23:04:55.035931 4066 ts_manager.cc:194] Registered new tserver with Master: 867d89767f304d9899b36f46739d613f (127.23.56.65:42235)
I20260810 23:04:55.036808 4066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:45845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:04:55.050916 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:38387/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43919
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:04:55 dist-test-slave-1xrh krb5kdc[3524](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 23:04:55.230393 4317 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:55.230667 4317 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:55.230703 4317 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:55.232538 4317 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:55.232595 4317 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:55.232635 4317 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:04:55.234781 4317 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.56.84:38387/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4317
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:55.235049 4317 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:55.235359 4317 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:55.242189 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:55.242216 4323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:55.242216 4324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:55.243336 4317 server_base.cc:1034] running on GCE node
I20260810 23:04:55.243556 4317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:55.243865 4317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:55.245074 4317 hybrid_clock.cc:648] HybridClock initialized: now 1786403095245047 us; error 40 us; skew 500 ppm
Aug 10 23:04:55 dist-test-slave-1xrh krb5kdc[3524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:55.248473 4317 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:04:55.249006 4317 webserver.cc:460] Webserver started at http://127.23.56.66:43169/ using document root <none> and password file <none>
I20260810 23:04:55.249267 4317 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:55.249349 4317 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:55.249496 4317 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:04:55 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:04:55.269169 4317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "c76397962c0644d68f84361be9ff8014"
format_stamp: "Formatted at 2026-08-10 23:04:55 on dist-test-slave-1xrh"
server_key: "61fed969c8d99ddc05c6549fd7a4f7a8"
server_key_iv: "be7feef32cebb6dfca50c34d40f9575e"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:55.269723 4317 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "c76397962c0644d68f84361be9ff8014"
format_stamp: "Formatted at 2026-08-10 23:04:55 on dist-test-slave-1xrh"
server_key: "61fed969c8d99ddc05c6549fd7a4f7a8"
server_key_iv: "be7feef32cebb6dfca50c34d40f9575e"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:55.274026 4317 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 23:04:55.290904 4335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:55.291172 4317 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 23:04:55.291297 4317 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "c76397962c0644d68f84361be9ff8014"
format_stamp: "Formatted at 2026-08-10 23:04:55 on dist-test-slave-1xrh"
server_key: "61fed969c8d99ddc05c6549fd7a4f7a8"
server_key_iv: "be7feef32cebb6dfca50c34d40f9575e"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:55.291410 4317 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:55.338030 4317 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:55.340785 4317 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:55.340973 4317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:55.341253 4317 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:55.341616 4317 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:55.341663 4317 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:55.341701 4317 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:55.341729 4317 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:55.348394 4317 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:39441
I20260810 23:04:55.348814 4317 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:04:55.352322 4471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:39441 every 8 connection(s)
I20260810 23:04:55.354734 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 4317
I20260810 23:04:55.354846 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:04:55.355012 23777 external_mini_cluster.cc:1428] Setting key 4bd4f343e2f3b7f62fec7eb5fd8edd82
Aug 10 23:04:55 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:04:55.363740 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.353078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:56021 (local address 127.23.56.126:43919)
0810 23:04:55.353132 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.353135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.353368 (+ 233us) server_negotiation.cc:407] Connection header received
0810 23:04:55.354089 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.354091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.354107 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.354139 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:55.354979 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.355818 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.357343 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.357668 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.357709 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.360429 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:55.360430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:55.360431 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:55.360439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:55.362183 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:55.362648 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:55.362650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:55.362650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:55.362705 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:55.362971 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:55.362972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:55.362972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:55.363100 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:55.363191 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.363548 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.363641 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:04:55.364311 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.353081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:43919 (local address 127.23.56.66:56021)
0810 23:04:55.353240 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:55.353281 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:55.353955 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:55.354321 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:55.354326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:55.354575 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:55.354846 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.354853 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.355955 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.355956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.357239 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.357242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.357803 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.357804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.357955 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.358615 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:55.358626 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:55.360319 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:55.362279 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:55.362286 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:55.362288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:55.362562 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:55.362771 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:55.362772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:55.362772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:55.362888 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:55.363239 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:55.363246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:55.363398 (+ 152us) client_negotiation.cc:769] Sending connection context
0810 23:04:55.363902 (+ 504us) client_negotiation.cc:240] Negotiation successful
0810 23:04:55.364073 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260810 23:04:55.364744 4472 heartbeater.cc:347] Connected to a master server at 127.23.56.126:43919
I20260810 23:04:55.364842 4472 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:55.365042 4472 heartbeater.cc:511] Master 127.23.56.126:43919 requested a full tablet report, sending...
I20260810 23:04:55.365415 4066 ts_manager.cc:194] Registered new tserver with Master: c76397962c0644d68f84361be9ff8014 (127.23.56.66:39441)
I20260810 23:04:55.365901 4066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:56021
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:04:55.410979 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:38387/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43919
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:33715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:04:55.542060 4477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:04:55.542423 4477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:04:55.542508 4477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:04:55.544196 4477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:04:55.544361 4477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:04:55.544596 4477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:04:55.546808 4477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:33715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.56.84:38387/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:43919
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:04:55.547333 4477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:04:55.547652 4477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:04:55.554263 4484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:04:55.554648 4483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:55.555671 4477 server_base.cc:1034] running on GCE node
W20260810 23:04:55.558082 4486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:04:55.558387 4477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:04:55.558655 4477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:04:55.559829 4477 hybrid_clock.cc:648] HybridClock initialized: now 1786403095559780 us; error 65 us; skew 500 ppm
Aug 10 23:04:55 dist-test-slave-1xrh krb5kdc[3524](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:04:55.563161 4477 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:04:55.563657 4477 webserver.cc:460] Webserver started at http://127.23.56.67:38739/ using document root <none> and password file <none>
I20260810 23:04:55.563941 4477 fs_manager.cc:359] Metadata directory not provided
I20260810 23:04:55.564000 4477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:04:55.564134 4477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:04:55 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:04:55.579164 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "0615e45289114915889be20feb3ba0e3"
format_stamp: "Formatted at 2026-08-10 23:04:55 on dist-test-slave-1xrh"
server_key: "d22b307833e79b463fd5cc81d7b53cca"
server_key_iv: "e34f3336b295d391ef6eb5ac704af26c"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:55.579655 4477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "0615e45289114915889be20feb3ba0e3"
format_stamp: "Formatted at 2026-08-10 23:04:55 on dist-test-slave-1xrh"
server_key: "d22b307833e79b463fd5cc81d7b53cca"
server_key_iv: "e34f3336b295d391ef6eb5ac704af26c"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:55.594367 4477 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.002s sys 0.000s
I20260810 23:04:55.609326 4496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:55.609576 4477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:04:55.609670 4477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "0615e45289114915889be20feb3ba0e3"
format_stamp: "Formatted at 2026-08-10 23:04:55 on dist-test-slave-1xrh"
server_key: "d22b307833e79b463fd5cc81d7b53cca"
server_key_iv: "e34f3336b295d391ef6eb5ac704af26c"
server_key_version: "kuduclusterkey@0"
I20260810 23:04:55.609755 4477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:04:55.621419 4477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:04:55.623664 4477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:04:55.623878 4477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:04:55.624116 4477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:04:55.624512 4477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:04:55.624598 4477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:55.624671 4477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:04:55.624728 4477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:04:55.634418 4477 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:35333
I20260810 23:04:55.634919 4477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:04:55.635633 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 4477
I20260810 23:04:55.635725 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:04:55.635913 23777 external_mini_cluster.cc:1428] Setting key f8011a5219cdb16c15ffe6abfd9f16e0
I20260810 23:04:55.637323 4624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:35333 every 8 connection(s)
Aug 10 23:04:55 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:04:55.649569 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.638201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:37671 (local address 127.23.56.126:43919)
0810 23:04:55.638244 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.638246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.638614 (+ 368us) server_negotiation.cc:407] Connection header received
0810 23:04:55.639315 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.639316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.639328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.639360 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:55.640191 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.640995 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.642493 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.642783 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.642834 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.645560 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:55.645561 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:55.645561 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:55.645569 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:55.647440 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:55.648496 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:55.648498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:55.648498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:55.648555 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:55.648833 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:55.648834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:55.648834 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:55.648968 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:55.649010 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.649387 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.649470 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:04:55.650302 4627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.637936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:43919 (local address 127.23.56.67:37671)
0810 23:04:55.638494 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:55.638537 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:55.639182 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:55.639546 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:55.639551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:55.639808 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:04:55.640079 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.640086 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.641112 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.641112 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.642385 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.642388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.642930 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.642931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.643079 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.643695 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:04:55.643706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:04:55.645445 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:04:55.648090 (+ 2645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:55.648097 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:55.648100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:55.648403 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:55.648624 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:04:55.648625 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:04:55.648625 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:04:55.648754 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:04:55.649054 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:04:55.649060 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:04:55.649213 (+ 153us) client_negotiation.cc:769] Sending connection context
0810 23:04:55.649900 (+ 687us) client_negotiation.cc:240] Negotiation successful
0810 23:04:55.650061 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":84,"threads_started":1}
I20260810 23:04:55.650979 4625 heartbeater.cc:347] Connected to a master server at 127.23.56.126:43919
I20260810 23:04:55.651068 4625 heartbeater.cc:464] Registering TS with master...
I20260810 23:04:55.651297 4625 heartbeater.cc:511] Master 127.23.56.126:43919 requested a full tablet report, sending...
I20260810 23:04:55.652717 4066 ts_manager.cc:194] Registered new tserver with Master: 0615e45289114915889be20feb3ba0e3 (127.23.56.67:35333)
I20260810 23:04:55.653206 4066 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:37671
I20260810 23:04:55.659197 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 23:04:55 dist-test-slave-1xrh krb5kdc[3524](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403095, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:04:55.675009 4312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.661108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.23.56.126:43919)
0810 23:04:55.661147 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.661149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.667363 (+ 6214us) server_negotiation.cc:407] Connection header received
0810 23:04:55.667471 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.667472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.667483 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.667514 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:04:55.668362 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.669181 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.670266 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.670567 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.670639 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.672127 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:04:55.672129 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:04:55.672129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:04:55.672137 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:04:55.673752 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:55.674148 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:55.674149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:55.674150 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:55.674207 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:04:55.674400 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:04:55.674401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:04:55.674401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:04:55.674533 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:04:55.674605 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.674831 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.674916 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:04:55.677043 4066 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:04:55.679543 4066 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:04:55.696019 4639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.688195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:39441 (local address 127.0.0.1:53734)
0810 23:04:55.689672 (+ 1477us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:55.689718 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:55.689788 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:55.690207 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:55.690210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:55.690213 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:55.690293 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.690294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.692089 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.692090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.693340 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.693343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.695675 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.695676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.695800 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.695816 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 23:04:55.695884 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:04:55.695937 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1441,"thread_start_us":87,"threads_started":1}
I20260810 23:04:55.700181 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.688288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.23.56.66:39441)
0810 23:04:55.689948 (+ 1660us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.689953 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.689963 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:04:55.690006 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.690009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.690026 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.690102 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:55.690526 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.691993 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.694625 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.695483 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.695554 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.695637 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.700043 (+ 4406us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.700089 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1619,"thread_start_us":90,"threads_started":1}
I20260810 23:04:55.700258 4637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.688442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:42235 (local address 127.0.0.1:49250)
0810 23:04:55.690393 (+ 1951us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:55.690402 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:55.690455 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:55.694221 (+ 3766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:55.694222 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:55.694225 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:55.694262 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.694263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.697891 (+ 3628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.697893 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.698980 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.698984 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.700001 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.700002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.700067 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.700071 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:04:55.700124 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 23:04:55.700178 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1936,"thread_start_us":64,"threads_started":1}
I20260810 23:04:55.700553 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.688737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.23.56.65:42235)
0810 23:04:55.693985 (+ 5248us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.693990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.694000 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:04:55.694036 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.694037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.694053 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.694127 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:55.696774 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.697777 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.699101 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.699831 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.699906 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.699962 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.700432 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.700476 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5207,"thread_start_us":67,"threads_started":1}
I20260810 23:04:55.701247 4229 tablet_service.cc:1467] Processing CreateTablet for tablet f4f36992e7f744fbadf8208e704de3f9 (DEFAULT_TABLE table=test-table [id=fa553535470a4c40836792978416dc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:55.701632 4229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f36992e7f744fbadf8208e704de3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:55.701539 4390 tablet_service.cc:1467] Processing CreateTablet for tablet f4f36992e7f744fbadf8208e704de3f9 (DEFAULT_TABLE table=test-table [id=fa553535470a4c40836792978416dc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:55.701874 4390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f36992e7f744fbadf8208e704de3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:55.702514 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.688962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:35333 (local address 127.0.0.1:37090)
0810 23:04:55.693816 (+ 4854us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:55.693826 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:55.693869 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:55.694688 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:55.694689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:55.694691 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:55.694729 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.694731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.698982 (+ 4251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.698983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.700900 (+ 1917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.700903 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.702267 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.702268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.702331 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.702336 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:04:55.702390 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 23:04:55.702443 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4837,"thread_start_us":82,"threads_started":1}
I20260810 23:04:55.702940 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.689235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.23.56.67:35333)
0810 23:04:55.694372 (+ 5137us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.694376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.694386 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:04:55.694422 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.694424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.694450 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.694523 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:55.697888 (+ 3365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.698883 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.701068 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.702072 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.702140 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.702230 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.702820 (+ 590us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.702864 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5084,"thread_start_us":102,"threads_started":1}
I20260810 23:04:55.703696 4547 tablet_service.cc:1467] Processing CreateTablet for tablet f4f36992e7f744fbadf8208e704de3f9 (DEFAULT_TABLE table=test-table [id=fa553535470a4c40836792978416dc2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:04:55.704044 4547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4f36992e7f744fbadf8208e704de3f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:04:55.705484 4644 tablet_bootstrap.cc:492] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: Bootstrap starting.
I20260810 23:04:55.705704 4643 tablet_bootstrap.cc:492] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f: Bootstrap starting.
I20260810 23:04:55.706588 4643 tablet_bootstrap.cc:654] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:55.706702 4644 tablet_bootstrap.cc:654] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:55.706961 4643 log.cc:824] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:55.707262 4644 log.cc:824] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:55.707688 4646 tablet_bootstrap.cc:492] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: Bootstrap starting.
I20260810 23:04:55.708714 4646 tablet_bootstrap.cc:654] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: Neither blocks nor log segments found. Creating new log.
I20260810 23:04:55.709017 4646 log.cc:824] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:04:55.710286 4644 tablet_bootstrap.cc:492] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: No bootstrap required, opened a new log
I20260810 23:04:55.710363 4644 ts_tablet_manager.cc:1397] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.001s
I20260810 23:04:55.710728 4643 tablet_bootstrap.cc:492] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f: No bootstrap required, opened a new log
I20260810 23:04:55.710850 4643 ts_tablet_manager.cc:1397] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260810 23:04:55.712453 4644 raft_consensus.cc:348] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } }
I20260810 23:04:55.712622 4644 raft_consensus.cc:374] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:55.712666 4644 raft_consensus.cc:729] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c76397962c0644d68f84361be9ff8014, State: Initialized, Role: FOLLOWER
I20260810 23:04:55.712776 4644 consensus_queue.cc:260] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } }
I20260810 23:04:55.713063 4644 ts_tablet_manager.cc:1428] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 23:04:55.713197 4472 heartbeater.cc:503] Master 127.23.56.126:43919 was elected leader, sending a full tablet report...
I20260810 23:04:55.714452 4643 raft_consensus.cc:348] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } }
I20260810 23:04:55.714675 4643 raft_consensus.cc:374] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:55.714736 4643 raft_consensus.cc:729] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867d89767f304d9899b36f46739d613f, State: Initialized, Role: FOLLOWER
I20260810 23:04:55.714913 4643 consensus_queue.cc:260] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } }
I20260810 23:04:55.715183 4643 ts_tablet_manager.cc:1428] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 23:04:55.715292 4309 heartbeater.cc:503] Master 127.23.56.126:43919 was elected leader, sending a full tablet report...
I20260810 23:04:55.716856 4646 tablet_bootstrap.cc:492] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: No bootstrap required, opened a new log
I20260810 23:04:55.716934 4646 ts_tablet_manager.cc:1397] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260810 23:04:55.718706 4646 raft_consensus.cc:348] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } }
I20260810 23:04:55.718858 4646 raft_consensus.cc:374] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:04:55.718887 4646 raft_consensus.cc:729] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0615e45289114915889be20feb3ba0e3, State: Initialized, Role: FOLLOWER
I20260810 23:04:55.719009 4646 consensus_queue.cc:260] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } }
I20260810 23:04:55.719269 4646 ts_tablet_manager.cc:1428] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:04:55.719318 4625 heartbeater.cc:503] Master 127.23.56.126:43919 was elected leader, sending a full tablet report...
W20260810 23:04:55.754120 4310 tablet.cc:2369] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:04:55.795480 4649 raft_consensus.cc:482] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:04:55.795641 4649 raft_consensus.cc:504] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } }
I20260810 23:04:55.796060 4649 leader_election.cc:288] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0615e45289114915889be20feb3ba0e3 (127.23.56.67:35333), 867d89767f304d9899b36f46739d613f (127.23.56.65:42235)
I20260810 23:04:55.800235 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.796535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:42235 (local address 127.23.56.66:42003)
0810 23:04:55.796659 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:55.796673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:55.796725 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:55.796914 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:55.796915 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:55.796917 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:55.796996 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.796998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.797912 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.797914 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.799085 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.799088 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.799988 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.799989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.800066 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.800070 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:04:55.800115 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 23:04:55.800154 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 23:04:55.800997 4640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.796685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:42003 (local address 127.23.56.65:42235)
0810 23:04:55.796731 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.796734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.796766 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:04:55.796798 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.796799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.796812 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.796858 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:55.797080 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.797805 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.799255 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.799867 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.799943 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.799954 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.800880 (+ 926us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.800926 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 23:04:55.801360 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f36992e7f744fbadf8208e704de3f9" candidate_uuid: "c76397962c0644d68f84361be9ff8014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "867d89767f304d9899b36f46739d613f" is_pre_election: true
I20260810 23:04:55.801569 4250 raft_consensus.cc:2454] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c76397962c0644d68f84361be9ff8014 in term 0.
I20260810 23:04:55.802544 4336 leader_election.cc:302] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 867d89767f304d9899b36f46739d613f, c76397962c0644d68f84361be9ff8014; no voters:
I20260810 23:04:55.802682 4649 raft_consensus.cc:2781] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:04:55.802733 4649 raft_consensus.cc:482] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:04:55.802757 4649 raft_consensus.cc:3037] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:55.803745 4649 raft_consensus.cc:504] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } }
I20260810 23:04:55.803916 4649 leader_election.cc:288] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [CANDIDATE]: Term 1 election: Requested vote from peers 0615e45289114915889be20feb3ba0e3 (127.23.56.67:35333), 867d89767f304d9899b36f46739d613f (127.23.56.65:42235)
I20260810 23:04:55.804368 4474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.796891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:35333 (local address 127.23.56.66:38079)
0810 23:04:55.800323 (+ 3432us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:55.800334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:04:55.800372 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:04:55.801348 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:04:55.801349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:04:55.801352 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:04:55.801383 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.801384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.802339 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.802340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.803389 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:04:55.803392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.804170 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.804171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:04:55.804228 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.804230 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:04:55.804274 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 23:04:55.804304 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3422,"thread_start_us":60,"threads_started":1}
I20260810 23:04:55.804559 4642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.797056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:38079 (local address 127.23.56.67:35333)
0810 23:04:55.797101 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.797103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.800592 (+ 3489us) server_negotiation.cc:407] Connection header received
0810 23:04:55.800624 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.800625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.800640 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.800687 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:04:55.801480 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.802203 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.803495 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.804057 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.804130 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.804138 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.804457 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.804500 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:04:55.804993 4569 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f36992e7f744fbadf8208e704de3f9" candidate_uuid: "c76397962c0644d68f84361be9ff8014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0615e45289114915889be20feb3ba0e3" is_pre_election: true
I20260810 23:04:55.805153 4569 raft_consensus.cc:2454] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c76397962c0644d68f84361be9ff8014 in term 0.
I20260810 23:04:55.805310 4568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f36992e7f744fbadf8208e704de3f9" candidate_uuid: "c76397962c0644d68f84361be9ff8014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0615e45289114915889be20feb3ba0e3"
I20260810 23:04:55.805377 4568 raft_consensus.cc:3037] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:55.806267 4568 raft_consensus.cc:2454] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c76397962c0644d68f84361be9ff8014 in term 1.
I20260810 23:04:55.806509 4339 leader_election.cc:302] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0615e45289114915889be20feb3ba0e3, c76397962c0644d68f84361be9ff8014; no voters:
I20260810 23:04:55.806504 4250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4f36992e7f744fbadf8208e704de3f9" candidate_uuid: "c76397962c0644d68f84361be9ff8014" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "867d89767f304d9899b36f46739d613f"
I20260810 23:04:55.806579 4250 raft_consensus.cc:3037] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:04:55.807484 4250 raft_consensus.cc:2454] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c76397962c0644d68f84361be9ff8014 in term 1.
I20260810 23:04:55.807706 4649 raft_consensus.cc:2781] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:04:55.807893 4649 raft_consensus.cc:686] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 1 LEADER]: Becoming Leader. State: Replica: c76397962c0644d68f84361be9ff8014, State: Running, Role: LEADER
I20260810 23:04:55.808007 4649 consensus_queue.cc:237] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } }
I20260810 23:04:55.808898 4065 catalog_manager.cc:5495] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 reported cstate change: term changed from 0 to 1, leader changed from <none> to c76397962c0644d68f84361be9ff8014 (127.23.56.66). New cstate: current_term: 1 leader_uuid: "c76397962c0644d68f84361be9ff8014" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c76397962c0644d68f84361be9ff8014" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 } health_report { overall_health: UNKNOWN } } }
I20260810 23:04:55.839185 4638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.835792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.23.56.66:39441)
0810 23:04:55.835840 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:04:55.835842 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:04:55.835851 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:04:55.835895 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:04:55.835895 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:04:55.835918 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:04:55.836008 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:04:55.836189 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.836930 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.838104 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:04:55.838420 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:04:55.838501 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:04:55.838731 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:04:55.838861 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:04:55.838959 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 23:04:55.839033 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 23:04:55.839058 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:04:55.841270 4568 raft_consensus.cc:1261] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [term 1 FOLLOWER]: Refusing update from remote peer c76397962c0644d68f84361be9ff8014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:55.841269 4250 raft_consensus.cc:1261] T f4f36992e7f744fbadf8208e704de3f9 P 867d89767f304d9899b36f46739d613f [term 1 FOLLOWER]: Refusing update from remote peer c76397962c0644d68f84361be9ff8014: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:04:55.841621 4649 consensus_queue.cc:1035] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "867d89767f304d9899b36f46739d613f" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 42235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:55.841744 4653 consensus_queue.cc:1035] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0615e45289114915889be20feb3ba0e3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 35333 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:04:55.845890 4659 mvcc.cc:205] Tried to move back new op lower bound from 7317107080562872320 to 7317107080430301184. Current Snapshot: MvccSnapshot[applied={T|T < 7317107080562872320}]
I20260810 23:04:55.846470 4658 mvcc.cc:205] Tried to move back new op lower bound from 7317107080562872320 to 7317107080430301184. Current Snapshot: MvccSnapshot[applied={T|T < 7317107080562872320}]
I20260810 23:04:55.849097 4065 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:55.849186 4065 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:04:55.850034 4065 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4734eeb31a249f199f917d1f66e63dc: Sending DeleteTablet for 3 replicas of tablet f4f36992e7f744fbadf8208e704de3f9
I20260810 23:04:55.850469 4390 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f36992e7f744fbadf8208e704de3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:55 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:53734
I20260810 23:04:55.850739 4229 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f36992e7f744fbadf8208e704de3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:55 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:49250
I20260810 23:04:55.851169 4547 tablet_service.cc:1514] Processing DeleteTablet for tablet f4f36992e7f744fbadf8208e704de3f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:04:55 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:37090
I20260810 23:04:55.851846 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 4158
I20260810 23:04:55.862949 4670 tablet_replica.cc:330] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: stopping tablet replica
I20260810 23:04:55.863165 4670 raft_consensus.cc:2229] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:04:55.863404 4670 raft_consensus.cc:2258] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:55.863742 4666 tablet_replica.cc:330] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: stopping tablet replica
I20260810 23:04:55.864063 4666 raft_consensus.cc:2229] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:04:55.864240 4666 raft_consensus.cc:2258] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:04:55.865489 4666 ts_tablet_manager.cc:1905] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:55.867691 4666 ts_tablet_manager.cc:1918] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:55.867846 4666 log.cc:1196] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/f4f36992e7f744fbadf8208e704de3f9
I20260810 23:04:55.868204 4666 ts_tablet_manager.cc:1939] T f4f36992e7f744fbadf8208e704de3f9 P c76397962c0644d68f84361be9ff8014: Deleting consensus metadata
I20260810 23:04:55.868731 4048 catalog_manager.cc:4841] TS c76397962c0644d68f84361be9ff8014 (127.23.56.66:39441): tablet f4f36992e7f744fbadf8208e704de3f9 (table test-table [id=fa553535470a4c40836792978416dc2b]) successfully deleted
I20260810 23:04:55.870209 4670 ts_tablet_manager.cc:1905] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:04:55.872207 4670 ts_tablet_manager.cc:1918] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:04:55.872372 4670 log.cc:1196] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/f4f36992e7f744fbadf8208e704de3f9
I20260810 23:04:55.873050 4670 ts_tablet_manager.cc:1939] T f4f36992e7f744fbadf8208e704de3f9 P 0615e45289114915889be20feb3ba0e3: Deleting consensus metadata
I20260810 23:04:55.873570 4052 catalog_manager.cc:4841] TS 0615e45289114915889be20feb3ba0e3 (127.23.56.67:35333): tablet f4f36992e7f744fbadf8208e704de3f9 (table test-table [id=fa553535470a4c40836792978416dc2b]) successfully deleted
W20260810 23:04:55.881683 4048 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:42235 (error 108)
I20260810 23:04:55.882436 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 4317
I20260810 23:04:55.891460 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 4477
I20260810 23:04:55.891597 4641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:04:55.891455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:42235 (local address 127.0.0.1:49252)
0810 23:04:55.891522 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 23:04:55.891562 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:42235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 23:04:55.893792 4048 catalog_manager.cc:4584] TS 867d89767f304d9899b36f46739d613f (127.23.56.65:42235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f4f36992e7f744fbadf8208e704de3f9: Network error: Client connection negotiation failed: client connection to 127.23.56.65:42235: connect: Connection refused (error 111)
I20260810 23:04:55.899480 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 4028
I20260810 23:04:55.906682 23777 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:04:56.251747 23777 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:04:56.251899 23777 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:04:56.596228 23777 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:04:56.596 UTC [3558] LOG: received smart shutdown request
2026-08-10 23:04:56.600 UTC [3558] LOG: background worker "logical replication launcher" (PID 3571) exited with exit code 1
2026-08-10 23:04:56.601 UTC [3565] LOG: shutting down
2026-08-10 23:04:56.786 UTC [3558] LOG: database system is shut down
2026-08-10T23:04:56Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47380 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:04:56 dist-test-slave-1xrh krb5kdc[4682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:04:56 dist-test-slave-1xrh krb5kdc[4682](info): set up 2 sockets
Aug 10 23:04:56 dist-test-slave-1xrh krb5kdc[4682](info): commencing operation
krb5kdc: starting...
W20260810 23:04:58.008621 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:04:58 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:04:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:04:58Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.56.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.56.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/rangeradmin_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/rangeradmin_127.23.56.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.56.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.56.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/rangerlookup_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/rangerlookup_127.23.56.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.56.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.56.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/HTTP_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/HTTP_127.23.56.84@KRBTEST.COM.keytab.
I20260810 23:04:58.182184 23777 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskQ_MaNL/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/postgres -l logfile start
2026-08-10 23:05:00.755 UTC [4718] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 23:05:00.755 UTC [4718] LOG: listening on IPv4 address "127.23.56.84", port 57479
2026-08-10 23:05:00.759 UTC [4718] LOG: listening on Unix socket "/tmp/.s.PGSQL.57479"
2026-08-10 23:05:00.781 UTC [4721] LOG: database system was shut down at 2026-08-10 23:04:59 UTC
2026-08-10 23:05:00.785 UTC [4718] LOG: database system is ready to accept connections
I20260810 23:05:01.804219 23777 mini_postgres.cc:96] Postgres bound to 57479
2026-08-10 23:05:01.811 UTC [4730] FATAL: database "slave" does not exist
127.23.56.84:57479 - accepting connections
I20260810 23:05:01.811944 23777 mini_ranger.cc:161] Starting Ranger...
I20260810 23:05:01.826702 23777 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 23:05:02.328179 23777 mini_ranger.cc:87] Created ranger Postgres database
I20260810 23:05:02.328266 23777 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-admin
2026-08-10 23:05:02,607 [I] DB FLAVOR :POSTGRES
2026-08-10 23:05:02,608 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:05:02,608 [I] Checking connection..
2026-08-10 23:05:02,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 23:05:02,858 [I] Checking connection passed.
2026-08-10 23:05:02,858 [I] --------- Verifying version history table ---------
2026-08-10 23:05:02,859 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:05:03,103 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:05:03,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:05:03,390 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 23:05:03,390 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 23:05:03,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 23:05:03.627 UTC [4802] WARNING: there is no transaction in progress
2026-08-10 23:05:03,634 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 23:05:03,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 23:05:03,861 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 23:05:03,861 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:05:03,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 23:05:04,120 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 23:05:04,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1xrh.c.gcp-upstream.internal','N') ;"
2026-08-10 23:05:04,642 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:05:04,904 [I] Table x_portal_user does not exist in database ranger
2026-08-10 23:05:04,904 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:05:05,161 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 23:05:05,162 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:05:05,421 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:05:05,666 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 23:05:05,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 23:05:06.641 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.656 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.673 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.689 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.709 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.730 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.799 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.806 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.821 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.831 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.841 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.855 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.864 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.871 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.877 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:06.884 UTC [4927] WARNING: there is no transaction in progress
2026-08-10 23:05:07,539 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 23:05:07,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 23:05:07,792 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 23:05:07,793 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 23:05:08,069 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 23:05:08,070 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 23:05:08,301 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 23:05:08,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 23:05:08,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 23:05:08,798 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1xrh.c.gcp-upstream.internal';"
2026-08-10 23:05:09,048 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 23:05:09,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 23:05:09,296 [I] DB_PATCHES have already been applied
I20260810 23:05:09.301628 23777 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-admin:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskQ_MaNL/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 23:05:09.301692 23777 mini_ranger.cc:193] Using host: 127.23.56.84
I20260810 23:05:09.305595 23777 mini_ranger.cc:227] Ranger admin URL: http://127.23.56.84:46565
Aug 10, 2026 11:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:05:09 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:05:10 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:05:10 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 23:05:10.482347 23777 mini_ranger.cc:160] Time spent starting Ranger: real 8.670s user 0.000s sys 0.010s
Aug 10, 2026 11:05:29 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:05:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Aug 10, 2026 11:05:29 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:05:29 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:05:30 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 23:05:31.008311 23777 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.56.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.56.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/rangerkms_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.56.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/rangerkms_127.23.56.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.56.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.56.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.56.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/HTTP_127.23.56.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.56.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/HTTP_127.23.56.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 23:05:31 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403131, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 23:05:31.108122 23777 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 23:05:31.117568 23777 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 23:05:32.021155 23777 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 23:05:32.021225 23777 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-kms
I20260810 23:05:32.021238 23777 mini_ranger_kms.cc:227] Using postgres at 127.23.56.84:57479
2026-08-10 23:05:32,193 [I] DB FLAVOR :POSTGRES
2026-08-10 23:05:32,194 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 23:05:32,194 [I] Checking connection
2026-08-10 23:05:32,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 23:05:32,462 [I] connection success
2026-08-10 23:05:32,462 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 23:05:32,462 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 23:05:32,462 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 23:05:32,712 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 23:05:32,712 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 23:05:32,712 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 23:05:32,712 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.56.84:57479/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 23:05:32,981 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 23:05:33.450868 23777 mini_ranger_kms.cc:310] Created kms service
I20260810 23:05:33.620175 23777 mini_ranger_kms.cc:326] Created kudu user
I20260810 23:05:33.663916 23777 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 23:05:33 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403131, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:34.046798 23777 mini_ranger_kms.cc:384] Added ranger policy
I20260810 23:05:34.047014 23777 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-kms:/tmp/dist-test-taskQ_MaNL/build/release/bin/postgresql.jar:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskQ_MaNL/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 23:05:34.047043 23777 mini_ranger_kms.cc:240] Using host: 127.23.56.84
I20260810 23:05:34.051795 23777 mini_ranger_kms.cc:276] Ranger KMS PID: 5137
I20260810 23:05:34.051869 23777 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.56.84:53631
Aug 10, 2026 11:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:05:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 23:05:36 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403136, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.56.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:36.474514 23777 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.366s user 0.004s sys 0.148s
I20260810 23:05:36.474618 23777 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 23:05:36.474666 23777 mini_ranger_kms.cc:401] 127.23.56.84:53631/kms/v1/keys
Aug 10 23:05:36 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403136, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.56.84@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
Aug 10 23:05:37 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403136, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.56.84@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
Aug 10 23:05:39 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:05:39.534783 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33761
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33761
--ranger_config_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:53631/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:39.616706 5197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:39.616935 5197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:39.616956 5197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:39.618140 5197 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:39.618185 5197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:39.618199 5197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:39.618212 5197 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:39.618224 5197 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:39.619927 5197 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.56.84:53631/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33761
--ranger_config_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:39.620184 5197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:39.620427 5197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:39.625619 5204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:39.625626 5206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:39.625641 5197 server_base.cc:1034] running on GCE node
W20260810 23:05:39.625823 5202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:39.626164 5197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:39.626489 5197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:39.627646 5197 hybrid_clock.cc:648] HybridClock initialized: now 1786403139627625 us; error 37 us; skew 500 ppm
Aug 10 23:05:39 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403139, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:39.630395 5197 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:05:39.630817 5197 webserver.cc:460] Webserver started at http://127.23.56.126:36029/ using document root <none> and password file <none>
I20260810 23:05:39.631084 5197 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:39.631170 5197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:39.631327 5197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:05:39 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403139, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:39.687382 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "571b21befd3346d9bbab30a84b1372d6"
format_stamp: "Formatted at 2026-08-10 23:05:39 on dist-test-slave-1xrh"
server_key: "f232d9763215c8ca102ec5dd55e0072c"
server_key_iv: "3806a8abad353d68b646a915634c11ca"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:39.687857 5197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "571b21befd3346d9bbab30a84b1372d6"
format_stamp: "Formatted at 2026-08-10 23:05:39 on dist-test-slave-1xrh"
server_key: "f232d9763215c8ca102ec5dd55e0072c"
server_key_iv: "3806a8abad353d68b646a915634c11ca"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:39.689613 5197 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 23:05:39.706135 5213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:39.706391 5197 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:39.706547 5197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "571b21befd3346d9bbab30a84b1372d6"
format_stamp: "Formatted at 2026-08-10 23:05:39 on dist-test-slave-1xrh"
server_key: "f232d9763215c8ca102ec5dd55e0072c"
server_key_iv: "3806a8abad353d68b646a915634c11ca"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:39.706697 5197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:39.724994 5197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:39.727381 5197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:39.727602 5197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:39.733168 5197 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33761
I20260810 23:05:39.733202 5276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33761 every 8 connection(s)
I20260810 23:05:39.733647 5197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:39.734411 5277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:39.736874 5277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Bootstrap starting.
I20260810 23:05:39.737504 5277 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:39.737823 5277 log.cc:824] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:39.738617 5277 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: No bootstrap required, opened a new log
I20260810 23:05:39.740859 5277 raft_consensus.cc:348] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } }
I20260810 23:05:39.740969 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 5197
I20260810 23:05:39.741035 5277 raft_consensus.cc:374] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:39.741081 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:05:39.741104 5277 raft_consensus.cc:729] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571b21befd3346d9bbab30a84b1372d6, State: Initialized, Role: FOLLOWER
I20260810 23:05:39.741261 23777 external_mini_cluster.cc:1428] Setting key d818f35c183fe2e03a04eff77fca2d06
I20260810 23:05:39.741281 5277 consensus_queue.cc:260] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } }
I20260810 23:05:39.741417 5277 raft_consensus.cc:388] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:39.741482 5277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:39.741549 5277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:39.742338 5277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } }
I20260810 23:05:39.742468 5277 leader_election.cc:302] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 571b21befd3346d9bbab30a84b1372d6; no voters:
I20260810 23:05:39.742715 5277 leader_election.cc:288] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:39.742774 5282 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:39.742986 5282 raft_consensus.cc:686] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 1 LEADER]: Becoming Leader. State: Replica: 571b21befd3346d9bbab30a84b1372d6, State: Running, Role: LEADER
I20260810 23:05:39.743070 5277 sys_catalog.cc:564] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:39.743166 5282 consensus_queue.cc:237] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } }
I20260810 23:05:39.743588 5282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "571b21befd3346d9bbab30a84b1372d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } } }
I20260810 23:05:39.743729 5282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:39.743934 5282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 571b21befd3346d9bbab30a84b1372d6. Latest consensus state: current_term: 1 leader_uuid: "571b21befd3346d9bbab30a84b1372d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } } }
I20260810 23:05:39.744047 5282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:39.744207 5290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:39.744719 5290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:39.745012 5277 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:05:39.750924 5290 catalog_manager.cc:1347] Generated new cluster ID: b43b1425f1b14b1992e591a6d0b9397c
I20260810 23:05:39.751055 5290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:05:39 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:39.758245 5281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:39.741826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59166 (local address 127.23.56.126:33761)
0810 23:05:39.742032 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:39.742041 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:39.742116 (+ 75us) server_negotiation.cc:407] Connection header received
0810 23:05:39.742235 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:39.742243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:39.742311 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:39.742451 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:39.743496 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:39.745448 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:39.747293 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:39.749203 (+ 1910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:39.752738 (+ 3535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:39.752943 (+ 205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:39.752948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:39.752948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:39.752961 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:39.755617 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:39.756083 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:39.756087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:39.756087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:39.756239 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:39.756543 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:39.756545 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:39.756545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:39.756950 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:39.757629 (+ 679us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:39.757716 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:05:39.757915 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260810 23:05:39.780124 5290 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:39.780915 5290 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:39.783646 5290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Generated new TSK 0
I20260810 23:05:39.783900 5290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:05:40.491556 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:53631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33761
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:40.584262 5323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:40.584494 5323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:40.584530 5323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:40.585683 5323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:40.585752 5323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:40.585806 5323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:05:40.587343 5323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.56.84:53631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:40.587615 5323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:40.587884 5323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:40.592924 5331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:40.592926 5333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:40.593372 5328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:40.593884 5323 server_base.cc:1034] running on GCE node
I20260810 23:05:40.594099 5323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:40.594367 5323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:40.595563 5323 hybrid_clock.cc:648] HybridClock initialized: now 1786403140595544 us; error 35 us; skew 500 ppm
Aug 10 23:05:40 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:40.598317 5323 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:05:40.598733 5323 webserver.cc:460] Webserver started at http://127.23.56.65:39211/ using document root <none> and password file <none>
I20260810 23:05:40.598985 5323 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:40.599068 5323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:40.599234 5323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:05:40 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:40.616743 5323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "e3dd59a2c70a416ea5b44d8c986d3275"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-1xrh"
server_key: "5e5967847427d25305ef52f20e15246c"
server_key_iv: "76b6c5fffc3d62fe41c8cd19d2d39892"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:40.617162 5323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "e3dd59a2c70a416ea5b44d8c986d3275"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-1xrh"
server_key: "5e5967847427d25305ef52f20e15246c"
server_key_iv: "76b6c5fffc3d62fe41c8cd19d2d39892"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:40.619287 5323 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 23:05:40.635313 5340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:40.635569 5323 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 23:05:40.635655 5323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "e3dd59a2c70a416ea5b44d8c986d3275"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-1xrh"
server_key: "5e5967847427d25305ef52f20e15246c"
server_key_iv: "76b6c5fffc3d62fe41c8cd19d2d39892"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:40.635752 5323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:40.651280 5323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:40.653726 5323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:40.653931 5323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:40.654186 5323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:40.654613 5323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:40.654716 5323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:40.654819 5323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:40.654896 5323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:40.661409 5323 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:37397
I20260810 23:05:40.661875 5323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:40.665521 5463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:37397 every 8 connection(s)
I20260810 23:05:40.669190 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 5323
I20260810 23:05:40.669298 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:05:40.669467 23777 external_mini_cluster.cc:1428] Setting key 74734dae5e0df8792fc578d8243f0e46
Aug 10 23:05:40 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:40.680145 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:40.666235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33761 (local address 127.23.56.65:36191)
0810 23:05:40.666441 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:40.666483 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:40.667103 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:40.667579 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:40.667583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:40.667825 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:40.668099 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:40.668105 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.670011 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.670012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:40.673469 (+ 3457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:40.673472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.674033 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.674034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:40.674171 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:40.674790 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:40.674801 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:40.676482 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:40.678559 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:40.678565 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:40.678568 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:40.678848 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:40.679058 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:40.679059 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:40.679060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:40.679198 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:40.679496 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:40.679502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:40.679657 (+ 155us) client_negotiation.cc:769] Sending connection context
0810 23:05:40.679755 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 23:05:40.679908 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":104,"threads_started":1}
I20260810 23:05:40.680806 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:40.666253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:36191 (local address 127.23.56.126:33761)
0810 23:05:40.667329 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:40.667331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:40.667340 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:05:40.667370 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:40.667371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:40.667384 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:40.667450 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:40.668214 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.669917 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.673573 (+ 3656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.673886 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.673953 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:40.676587 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:40.676588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:40.676589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:40.676596 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:40.678464 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:40.678929 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:40.678930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:40.678931 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:40.678994 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:40.679267 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:40.679268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:40.679268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:40.679411 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:40.679470 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:40.680618 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0810 23:05:40.680709 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1032,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
I20260810 23:05:40.681481 5464 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33761
I20260810 23:05:40.681584 5464 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:40.681772 5464 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
I20260810 23:05:40.682428 5231 ts_manager.cc:194] Registered new tserver with Master: e3dd59a2c70a416ea5b44d8c986d3275 (127.23.56.65:37397)
I20260810 23:05:40.683100 5231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:36191
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:05:40.702363 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:53631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33761
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:40.815014 5473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:40.815295 5473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:40.815379 5473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:40.816730 5473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:40.816828 5473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:40.816901 5473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:05:40.818638 5473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.56.84:53631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:40.818904 5473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:40.819211 5473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:40.824180 5481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:40.824245 5484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:40.824446 5482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:40.825129 5473 server_base.cc:1034] running on GCE node
I20260810 23:05:40.825340 5473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:40.825598 5473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:40.826751 5473 hybrid_clock.cc:648] HybridClock initialized: now 1786403140826731 us; error 38 us; skew 500 ppm
Aug 10 23:05:40 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:40.829591 5473 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:05:40.830152 5473 webserver.cc:460] Webserver started at http://127.23.56.66:43435/ using document root <none> and password file <none>
I20260810 23:05:40.830436 5473 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:40.830508 5473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:40.830646 5473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:05:40 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:40.850044 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "40dd6044f2034ea38b2ffa7190dfbda5"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-1xrh"
server_key: "5adea5ff6afc12fd56fca242349e4d10"
server_key_iv: "4c7528e06e9104da956d68ff200116a0"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:40.850497 5473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "40dd6044f2034ea38b2ffa7190dfbda5"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-1xrh"
server_key: "5adea5ff6afc12fd56fca242349e4d10"
server_key_iv: "4c7528e06e9104da956d68ff200116a0"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:40.852550 5473 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 23:05:40.866806 5491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:40.866981 5473 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260810 23:05:40.867070 5473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "40dd6044f2034ea38b2ffa7190dfbda5"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-1xrh"
server_key: "5adea5ff6afc12fd56fca242349e4d10"
server_key_iv: "4c7528e06e9104da956d68ff200116a0"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:40.867202 5473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 10 23:05:40 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:40.887465 5473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:40.889925 5473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:40.890130 5473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:40.890383 5473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:40.890785 5473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:40.890867 5473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:40.890941 5473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:40.891010 5473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:40.897732 5473 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:46741
I20260810 23:05:40.898181 5473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:40.899343 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 5473
I20260810 23:05:40.899446 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:05:40.899652 23777 external_mini_cluster.cc:1428] Setting key 70f48fd540d638d77cd688681eb4673a
I20260810 23:05:40.900894 5621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:46741 every 8 connection(s)
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Aug 10 23:05:40 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403140, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:40.922034 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:40.901543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:52229 (local address 127.23.56.126:33761)
0810 23:05:40.901600 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:40.901603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:40.902591 (+ 988us) server_negotiation.cc:407] Connection header received
0810 23:05:40.906160 (+ 3569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:40.906161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:40.906178 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:40.906210 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:40.907066 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.907888 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.913504 (+ 5616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.913817 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.913881 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:40.918499 (+ 4618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:40.918501 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:40.918501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:40.918509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:40.920371 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:40.920911 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:40.920912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:40.920913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:40.920982 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:40.921276 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:40.921276 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:40.921277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:40.921413 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:40.921489 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:40.921839 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 23:05:40.921932 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 23:05:40.922590 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:40.901537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33761 (local address 127.23.56.66:52229)
0810 23:05:40.902460 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:40.902507 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:40.906032 (+ 3525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:40.906426 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:40.906431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:40.906677 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:40.906948 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:40.906954 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.908008 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.908009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:40.909256 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:40.909259 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:40.913992 (+ 4733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:40.913993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:40.914131 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:40.914737 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:40.914748 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:40.918378 (+ 3630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:40.920472 (+ 2094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:40.920480 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:40.920482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:40.920796 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:40.921064 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:40.921065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:40.921066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:40.921204 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:40.921523 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:40.921529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:40.921687 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 23:05:40.922170 (+ 483us) client_negotiation.cc:240] Negotiation successful
0810 23:05:40.922354 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":873,"thread_start_us":77,"threads_started":1}
I20260810 23:05:40.923092 5622 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33761
I20260810 23:05:40.923213 5622 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:40.923405 5622 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
I20260810 23:05:40.923753 5231 ts_manager.cc:194] Registered new tserver with Master: 40dd6044f2034ea38b2ffa7190dfbda5 (127.23.56.66:46741)
I20260810 23:05:40.924263 5231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:52229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:05:40.942241 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:53631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33761
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:41.058233 5627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:41.058511 5627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:41.058548 5627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:41.060238 5627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:41.060293 5627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:41.060331 5627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:05:41.062451 5627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.56.84:53631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:41.062740 5627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:41.063089 5627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:41.069670 5637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.069718 5635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.069832 5634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:41.069960 5627 server_base.cc:1034] running on GCE node
I20260810 23:05:41.070261 5627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:41.070513 5627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:41.071692 5627 hybrid_clock.cc:648] HybridClock initialized: now 1786403141071661 us; error 47 us; skew 500 ppm
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:41.074975 5627 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:05:41.075552 5627 webserver.cc:460] Webserver started at http://127.23.56.67:33287/ using document root <none> and password file <none>
I20260810 23:05:41.075855 5627 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:41.075927 5627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:41.076077 5627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:41.091312 5627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "35a617764aca43e3b9daae1f8d3b3d61"
format_stamp: "Formatted at 2026-08-10 23:05:41 on dist-test-slave-1xrh"
server_key: "38857ed621836e822e1744a1bb74c01c"
server_key_iv: "76da8cf66786d08108f01cd50490d83c"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:41.091948 5627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "35a617764aca43e3b9daae1f8d3b3d61"
format_stamp: "Formatted at 2026-08-10 23:05:41 on dist-test-slave-1xrh"
server_key: "38857ed621836e822e1744a1bb74c01c"
server_key_iv: "76da8cf66786d08108f01cd50490d83c"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:41.103317 5627 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.002s sys 0.000s
I20260810 23:05:41.119439 5644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.119676 5627 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:41.119757 5627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "35a617764aca43e3b9daae1f8d3b3d61"
format_stamp: "Formatted at 2026-08-10 23:05:41 on dist-test-slave-1xrh"
server_key: "38857ed621836e822e1744a1bb74c01c"
server_key_iv: "76da8cf66786d08108f01cd50490d83c"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:41.119875 5627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:41.136313 5627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:41.138782 5627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:41.138967 5627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:41.139225 5627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:41.139616 5627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:41.139683 5627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.139729 5627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:41.139761 5627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.145813 5627 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:39083
I20260810 23:05:41.146188 5627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:41.149391 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 5627
I20260810 23:05:41.149529 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:05:41.149724 23777 external_mini_cluster.cc:1428] Setting key 12af54fc0ba944a8043d6e8b915eea36
I20260810 23:05:41.155890 5775 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:39083 every 8 connection(s)
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:41.165915 5468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:41.156617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:45805 (local address 127.23.56.126:33761)
0810 23:05:41.156668 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:41.156671 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:41.156965 (+ 294us) server_negotiation.cc:407] Connection header received
0810 23:05:41.157629 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:41.157630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:41.157641 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:41.157681 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:41.158411 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.158936 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.160409 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.160601 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.160656 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:41.162901 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:41.162903 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:41.162903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:41.162911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:41.164481 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:41.164912 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:41.164914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:41.164914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:41.164973 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:41.165241 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:41.165242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:41.165242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:41.165369 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:41.165445 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:41.165729 (+ 284us) server_negotiation.cc:299] Negotiation successful
0810 23:05:41.165825 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 23:05:41.166134 5778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:41.156566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33761 (local address 127.23.56.67:45805)
0810 23:05:41.156798 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:41.156848 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:41.157501 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:41.157774 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:41.157779 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:41.158032 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:41.158326 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:41.158332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.159036 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.159037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:41.160327 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:41.160330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.160695 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.160696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:41.160832 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:41.161354 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:41.161364 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:41.162801 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:41.164586 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:41.164593 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:41.164595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:41.164831 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:41.165051 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:41.165052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:41.165052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:41.165165 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:41.165437 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:41.165443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:41.165586 (+ 143us) client_negotiation.cc:769] Sending connection context
0810 23:05:41.165693 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 23:05:41.165850 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
I20260810 23:05:41.166548 5776 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33761
I20260810 23:05:41.166642 5776 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:41.166846 5776 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
I20260810 23:05:41.167243 5231 ts_manager.cc:194] Registered new tserver with Master: 35a617764aca43e3b9daae1f8d3b3d61 (127.23.56.67:39083)
I20260810 23:05:41.167624 5231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:45805
I20260810 23:05:41.175428 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:41.176156 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 5323
I20260810 23:05:41.192610 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 5473
I20260810 23:05:41.199448 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 5627
I20260810 23:05:41.205717 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 5197
I20260810 23:05:41.213820 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33761
--webserver_interface=127.23.56.126
--webserver_port=36029
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33761
--ranger_config_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:53631/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.56.84:53631/kms with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:41.296223 5779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:41.296413 5779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:41.296433 5779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:41.297503 5779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:41.297542 5779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:41.297555 5779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:41.297566 5779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:41.297577 5779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:41.299072 5779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.56.84:53631/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33761
--ranger_config_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=36029
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:41.299327 5779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:41.299546 5779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:41.304369 5784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.304476 5787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.304476 5785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:41.304603 5779 server_base.cc:1034] running on GCE node
I20260810 23:05:41.304878 5779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:41.305186 5779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:41.306448 5779 hybrid_clock.cc:648] HybridClock initialized: now 1786403141306430 us; error 46 us; skew 500 ppm
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:41.309211 5779 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:05:41.309782 5779 webserver.cc:460] Webserver started at http://127.23.56.126:36029/ using document root <none> and password file <none>
I20260810 23:05:41.310096 5779 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:41.310173 5779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:41.395407 5779 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:41.396234 5802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.396445 5779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:41.396546 5779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "571b21befd3346d9bbab30a84b1372d6"
format_stamp: "Formatted at 2026-08-10 23:05:39 on dist-test-slave-1xrh"
server_key: "f232d9763215c8ca102ec5dd55e0072c"
server_key_iv: "3806a8abad353d68b646a915634c11ca"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:41.397003 5779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:41.412735 5779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:41.414765 5779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:41.414875 5779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:41.419198 5779 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33761
I20260810 23:05:41.419222 5862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33761 every 8 connection(s)
I20260810 23:05:41.419648 5779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:41.420068 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 5779
I20260810 23:05:41.420387 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:53631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:37397
--local_ip_for_outbound_sockets=127.23.56.65
--tserver_master_addrs=127.23.56.126:33761
--webserver_port=39211
--webserver_interface=127.23.56.65
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.56.84:53631/kms with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:41.421408 5863 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:05:41.422447 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Bootstrap starting.
I20260810 23:05:41.430585 5863 log.cc:824] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:41.432366 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:05:41.432695 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Bootstrap complete.
I20260810 23:05:41.435241 5863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } }
I20260810 23:05:41.435482 5863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 571b21befd3346d9bbab30a84b1372d6, State: Initialized, Role: FOLLOWER
I20260810 23:05:41.435655 5863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } }
I20260810 23:05:41.435717 5863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:41.435766 5863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:41.435815 5863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:05:41.436822 5863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } }
I20260810 23:05:41.436936 5863 leader_election.cc:302] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 571b21befd3346d9bbab30a84b1372d6; no voters:
I20260810 23:05:41.437134 5863 leader_election.cc:288] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:05:41.437175 5867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:05:41.437318 5867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [term 2 LEADER]: Becoming Leader. State: Replica: 571b21befd3346d9bbab30a84b1372d6, State: Running, Role: LEADER
I20260810 23:05:41.437399 5863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:41.437474 5867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } }
I20260810 23:05:41.437834 5870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "571b21befd3346d9bbab30a84b1372d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } } }
I20260810 23:05:41.437835 5867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 571b21befd3346d9bbab30a84b1372d6. Latest consensus state: current_term: 2 leader_uuid: "571b21befd3346d9bbab30a84b1372d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "571b21befd3346d9bbab30a84b1372d6" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33761 } } }
I20260810 23:05:41.438112 5867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:41.438097 5870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:41.438334 5878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:41.438895 5878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:41.439069 5878 catalog_manager.cc:1259] Loaded cluster ID: b43b1425f1b14b1992e591a6d0b9397c
I20260810 23:05:41.439159 5878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:41.439247 5863 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 23:05:41.440395 5878 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:41.440572 5878 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Loaded TSK: 0
I20260810 23:05:41.440784 5878 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:05:41.530874 5865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:41.531253 5865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:41.531365 5865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:41.533085 5865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:41.533149 5865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:41.533190 5865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:05:41.535341 5865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.56.84:53631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:37397
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=39211
--tserver_master_addrs=127.23.56.126:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:41.535597 5865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:41.535892 5865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:41.542114 5903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.542309 5906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.542510 5904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:41.542709 5865 server_base.cc:1034] running on GCE node
I20260810 23:05:41.542903 5865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:41.543179 5865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:41.544445 5865 hybrid_clock.cc:648] HybridClock initialized: now 1786403141544413 us; error 54 us; skew 500 ppm
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:41.547799 5865 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:05:41.548309 5865 webserver.cc:460] Webserver started at http://127.23.56.65:39211/ using document root <none> and password file <none>
I20260810 23:05:41.548590 5865 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:41.548667 5865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:41.571516 5865 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:41.572629 5914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.572832 5865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 23:05:41.572918 5865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "e3dd59a2c70a416ea5b44d8c986d3275"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-1xrh"
server_key: "5e5967847427d25305ef52f20e15246c"
server_key_iv: "76b6c5fffc3d62fe41c8cd19d2d39892"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:41.573357 5865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:41.595809 5865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:41.598414 5865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:41.598641 5865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:41.598927 5865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:41.599385 5865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:41.599480 5865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.599586 5865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:41.599659 5865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.606094 5865 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:37397
I20260810 23:05:41.606640 5865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:41.607789 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 5865
I20260810 23:05:41.608114 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:53631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:46741
--local_ip_for_outbound_sockets=127.23.56.66
--tserver_master_addrs=127.23.56.126:33761
--webserver_port=43435
--webserver_interface=127.23.56.66
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.56.84:53631/kms with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:41.609764 6046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:37397 every 8 connection(s)
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:41.634388 6050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:41.610398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33761 (local address 127.23.56.65:46947)
0810 23:05:41.611269 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:41.611313 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:41.611944 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:41.619240 (+ 7296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:41.619246 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:41.619512 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:41.619834 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:41.619841 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.622214 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.622215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:41.623547 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:41.623551 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.624582 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.624583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:41.624729 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:41.625501 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:41.625512 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:41.627304 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:41.632500 (+ 5196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:41.632508 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:41.632511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:41.632804 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:41.633066 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:41.633066 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:41.633067 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:41.633189 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:41.633717 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:41.633724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:41.633878 (+ 154us) client_negotiation.cc:769] Sending connection context
0810 23:05:41.633983 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 23:05:41.634150 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"thread_start_us":90,"threads_started":1}
I20260810 23:05:41.634961 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:41.610428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:46947 (local address 127.23.56.126:33761)
0810 23:05:41.618477 (+ 8049us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:41.618489 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:41.618541 (+ 52us) server_negotiation.cc:407] Connection header received
0810 23:05:41.618708 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:41.618717 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:41.618783 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:41.619012 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:41.620621 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.622077 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.623654 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.624331 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.624456 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:41.627467 (+ 3011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:41.627471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:41.627472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:41.627484 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:41.632376 (+ 4892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:41.632893 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:41.632897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:41.632897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:41.632999 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:41.633263 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:41.633264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:41.633265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:41.633596 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:41.633684 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:41.634630 (+ 946us) server_negotiation.cc:299] Negotiation successful
0810 23:05:41.634761 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7970,"thread_start_us":95,"threads_started":1}
I20260810 23:05:41.635833 6047 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33761
I20260810 23:05:41.635928 6047 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:41.636109 6047 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
W20260810 23:05:41.636982 6047 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:33761 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:05:41.637045 6047 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:41.637100 6047 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
I20260810 23:05:41.637465 6047 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:41.637531 6047 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
W20260810 23:05:41.637823 6047 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:05:41.722078 6049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:41.722319 6049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:41.722361 6049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:41.723719 6049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:41.723759 6049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:41.723800 6049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:05:41.725342 6049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.56.84:53631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:46741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=43435
--tserver_master_addrs=127.23.56.126:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:41.725581 6049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:41.725821 6049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:41.731230 6057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.731243 6060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.731513 6058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:41.732044 6049 server_base.cc:1034] running on GCE node
I20260810 23:05:41.732247 6049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:41.732488 6049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:41.733669 6049 hybrid_clock.cc:648] HybridClock initialized: now 1786403141733654 us; error 34 us; skew 500 ppm
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:41.736931 6049 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:05:41.737457 6049 webserver.cc:460] Webserver started at http://127.23.56.66:43435/ using document root <none> and password file <none>
I20260810 23:05:41.737706 6049 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:41.737808 6049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:41.754200 6049 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:41.755378 6069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.755597 6049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 23:05:41.755692 6049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "40dd6044f2034ea38b2ffa7190dfbda5"
format_stamp: "Formatted at 2026-08-10 23:05:40 on dist-test-slave-1xrh"
server_key: "5adea5ff6afc12fd56fca242349e4d10"
server_key_iv: "4c7528e06e9104da956d68ff200116a0"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:41.756157 6049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:41.768430 6049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:41.770895 6049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:41.771109 6049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:41.771526 6049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:41.771946 6049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:41.772042 6049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.772116 6049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:41.772177 6049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.778908 6049 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:46741
I20260810 23:05:41.779457 6049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:41.787463 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 6049
I20260810 23:05:41.787799 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.56.84:53631/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:39083
--local_ip_for_outbound_sockets=127.23.56.67
--tserver_master_addrs=127.23.56.126:33761
--webserver_port=33287
--webserver_interface=127.23.56.67
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.56.84:53631/kms with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:41.791590 6201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:46741 every 8 connection(s)
I20260810 23:05:41.794787 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:41.780740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:38393 (local address 127.23.56.126:33761)
0810 23:05:41.780786 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:41.780788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:41.780801 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:05:41.782127 (+ 1326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:41.782130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:41.782148 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:41.782223 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:41.783000 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.783822 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.785320 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.785637 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.785688 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:41.790940 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:41.790942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:41.790942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:41.790952 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:41.792741 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:41.793556 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:41.793557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:41.793558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:41.793626 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:41.793948 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:41.793949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:41.793950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:41.794103 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:41.794209 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:41.794582 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 23:05:41.794691 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:05:41.795614 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:41.780411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33761 (local address 127.23.56.66:38393)
0810 23:05:41.780663 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:41.780714 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:41.781983 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:41.782321 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:41.782326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:41.782578 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:41.782870 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:41.782877 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.783951 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.783952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:41.785211 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:41.785214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:41.785812 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:41.785813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:41.785966 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:41.786597 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:41.786608 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:41.790781 (+ 4173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:41.793111 (+ 2330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:41.793118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:41.793121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:41.793444 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:41.793711 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:41.793712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:41.793712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:41.793855 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:41.794217 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:41.794225 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:41.794397 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 23:05:41.795193 (+ 796us) client_negotiation.cc:240] Negotiation successful
0810 23:05:41.795377 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":101,"threads_started":1}
I20260810 23:05:41.796330 6202 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33761
I20260810 23:05:41.796427 6202 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:41.796624 6202 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
W20260810 23:05:41.797376 6202 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:33761 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:05:41.797448 6202 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:41.797528 6202 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
I20260810 23:05:41.797953 6202 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:41.798033 6202 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
W20260810 23:05:41.798373 6202 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:05:41.920425 6205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:41.920789 6205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:41.920887 6205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:41.922672 6205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:41.922791 6205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:41.922883 6205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:05:41.925148 6205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.56.84:53631/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:39083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=33287
--tserver_master_addrs=127.23.56.126:33761
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:41.925681 6205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:41.926031 6205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:41.932816 6210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:41.933821 6205 server_base.cc:1034] running on GCE node
W20260810 23:05:41.935109 6211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:41.936614 6213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:41.936945 6205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:41.937232 6205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:41.952863 6205 hybrid_clock.cc:648] HybridClock initialized: now 1786403141952829 us; error 48 us; skew 500 ppm
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:41.956312 6205 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:05:41.956897 6205 webserver.cc:460] Webserver started at http://127.23.56.67:33287/ using document root <none> and password file <none>
I20260810 23:05:41.957199 6205 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:41.957281 6205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 23:05:41 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:41.976598 6205 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:41.977686 6222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.977962 6205 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 23:05:41.978044 6205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "35a617764aca43e3b9daae1f8d3b3d61"
format_stamp: "Formatted at 2026-08-10 23:05:41 on dist-test-slave-1xrh"
server_key: "38857ed621836e822e1744a1bb74c01c"
server_key_iv: "76da8cf66786d08108f01cd50490d83c"
server_key_version: "kuduclusterkey@0"
I20260810 23:05:41.978477 6205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:41.990206 6205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:41.992486 6205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:41.992671 6205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:41.992926 6205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:41.993314 6205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:41.993397 6205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:41.993465 6205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:41.993522 6205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:42.000041 6205 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:39083
I20260810 23:05:42.000797 6205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:42.008256 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 6205
I20260810 23:05:42.003547 6357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:39083 every 8 connection(s)
Aug 10 23:05:42 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403141, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:42.015893 6052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:42.004456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:48951 (local address 127.23.56.126:33761)
0810 23:05:42.004503 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:42.004505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:42.004835 (+ 330us) server_negotiation.cc:407] Connection header received
0810 23:05:42.005538 (+ 703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:42.005540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:42.005554 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:42.005584 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:42.006379 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:42.007203 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:42.009429 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:42.009743 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:42.009788 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:42.012617 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:42.012619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:42.012620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:42.012627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:42.014311 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:42.014791 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:42.014793 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:42.014793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:42.014857 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:42.015198 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:42.015199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:42.015200 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:42.015327 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:42.015368 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:42.015707 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 23:05:42.015794 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:05:42.016585 6360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:42.004162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33761 (local address 127.23.56.67:48951)
0810 23:05:42.004709 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:42.004753 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:42.005409 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:42.005768 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:42.005772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:42.006007 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:42.006271 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:42.006276 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:42.007363 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:42.007364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:42.009323 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:42.009326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:42.009933 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:42.009934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:42.010069 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:42.010681 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:42.010692 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:42.012511 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:42.014406 (+ 1895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:42.014413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:42.014415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:42.014695 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:42.014946 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:42.014947 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:42.014947 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:42.015082 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:42.015414 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:42.015420 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:42.015567 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 23:05:42.016217 (+ 650us) client_negotiation.cc:240] Negotiation successful
0810 23:05:42.016374 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":94,"threads_started":1}
I20260810 23:05:42.017270 6358 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33761
I20260810 23:05:42.017359 6358 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:42.017542 6358 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
W20260810 23:05:42.018222 6358 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:33761 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 23:05:42.018285 6358 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:42.018342 6358 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
I20260810 23:05:42.018662 6358 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:42.018728 6358 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
W20260810 23:05:42.019032 6358 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:05:42.026546 6361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:42.008673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59174 (local address 127.23.56.126:33761)
0810 23:05:42.010333 (+ 1660us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:42.010335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:42.019331 (+ 8996us) server_negotiation.cc:407] Connection header received
0810 23:05:42.019420 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:42.019421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:42.019432 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:42.019502 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:42.020122 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:42.020776 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:42.021543 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:42.021803 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:42.021887 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:42.023284 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:42.023285 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:42.023286 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:42.023293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:42.025060 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:42.025462 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:42.025463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:42.025463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:42.025519 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:42.025703 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:42.025704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:42.025704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:42.025817 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:42.026324 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:42.026364 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 23:05:42.026444 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1632,"thread_start_us":73,"threads_started":1}
Aug 10 23:05:42 dist-test-slave-1xrh krb5kdc[4682](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786403142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:42.638032 6047 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:42.638214 6047 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
I20260810 23:05:42.638619 5822 ts_manager.cc:194] Registered new tserver with Master: e3dd59a2c70a416ea5b44d8c986d3275 (127.23.56.65:37397)
I20260810 23:05:42.639730 5822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:46947
I20260810 23:05:42.798573 6202 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:42.798728 6202 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
I20260810 23:05:42.799093 5822 ts_manager.cc:194] Registered new tserver with Master: 40dd6044f2034ea38b2ffa7190dfbda5 (127.23.56.66:46741)
I20260810 23:05:42.799516 5822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:38393
Aug 10 23:05:42 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:43.019217 6358 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:43.019389 6358 heartbeater.cc:511] Master 127.23.56.126:33761 requested a full tablet report, sending...
I20260810 23:05:43.019775 5822 ts_manager.cc:194] Registered new tserver with Master: 35a617764aca43e3b9daae1f8d3b3d61 (127.23.56.67:39083)
I20260810 23:05:43.020313 5822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:48951
I20260810 23:05:43.027186 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:43.042009 6387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.028310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59180 (local address 127.23.56.126:33761)
0810 23:05:43.028731 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:43.028733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:43.028740 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:05:43.028762 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:43.028763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:43.028775 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:43.029547 (+ 772us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:43.029585 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.030264 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.031230 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.031521 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.032228 (+ 707us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.038533 (+ 6305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:43.038534 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:43.038535 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:43.038544 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:43.040471 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:43.040868 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:43.040870 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:43.040870 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:43.040934 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:43.041137 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:43.041138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:43.041138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:43.041268 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:43.041776 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:43.041822 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 23:05:43.041910 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":65,"threads_started":1}
I20260810 23:05:43.044375 5822 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:05:43.046810 5822 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:05:43.055264 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.050089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:46741 (local address 127.0.0.1:59024)
0810 23:05:43.050632 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:43.050673 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:43.050742 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:43.050984 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:43.050988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:43.050991 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:43.051060 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.051062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.052384 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.052385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.053914 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.053917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.054936 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.054937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.055022 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.055035 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:05:43.055103 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 23:05:43.055180 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":107,"threads_started":1}
I20260810 23:05:43.055830 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.050371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59024 (local address 127.23.56.66:46741)
0810 23:05:43.050561 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:43.050567 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:43.050826 (+ 259us) server_negotiation.cc:407] Connection header received
0810 23:05:43.050864 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:43.050866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:43.050882 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:43.050954 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:43.051208 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.052269 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.054012 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.054761 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.054835 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.054890 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:43.055701 (+ 811us) server_negotiation.cc:299] Negotiation successful
0810 23:05:43.055749 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":56,"threads_started":1}
I20260810 23:05:43.056713 6393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.050338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:39083 (local address 127.0.0.1:38762)
0810 23:05:43.052876 (+ 2538us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:43.052884 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:43.052931 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:43.053223 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:43.053224 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:43.053226 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:43.053257 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.053258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.054421 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.054422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.055500 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.055503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.056466 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.056467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.056547 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.056552 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:05:43.056599 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:05:43.056637 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2523,"mutex_wait_us":29,"thread_start_us":68,"threads_started":1}
I20260810 23:05:43.056782 6123 tablet_service.cc:1467] Processing CreateTablet for tablet dcbf9c4cc00c4c368797e63620632176 (DEFAULT_TABLE table=test-table [id=ba3483bf2a9e487d8ac7df29d2066259]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:43.056938 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.050445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.23.56.67:39083)
0810 23:05:43.051108 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:43.051113 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:43.052988 (+ 1875us) server_negotiation.cc:407] Connection header received
0810 23:05:43.053027 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:43.053029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:43.053043 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:43.053134 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:43.053331 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.054344 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.055591 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.056310 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.056388 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.056434 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:43.056816 (+ 382us) server_negotiation.cc:299] Negotiation successful
0810 23:05:43.056860 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":631,"thread_start_us":71,"threads_started":1}
I20260810 23:05:43.057156 6123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbf9c4cc00c4c368797e63620632176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:43.057245 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.050089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:37397 (local address 127.0.0.1:42532)
0810 23:05:43.052687 (+ 2598us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:43.052710 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:43.052789 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:43.053034 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:43.053035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:43.053038 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:43.053093 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.053113 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.054328 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.054329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.055614 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.055616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.056977 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.056990 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.057058 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.057062 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:05:43.057111 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 23:05:43.057153 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2582,"mutex_wait_us":1,"thread_start_us":101,"threads_started":1}
I20260810 23:05:43.057552 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.050547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42532 (local address 127.23.56.65:37397)
0810 23:05:43.050706 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:43.050711 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:43.052786 (+ 2075us) server_negotiation.cc:407] Connection header received
0810 23:05:43.052853 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:43.052856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:43.052872 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:43.052946 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:43.053189 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.054224 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.055693 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.056831 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.056893 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.056942 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:43.057448 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 23:05:43.057488 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
I20260810 23:05:43.057616 6279 tablet_service.cc:1467] Processing CreateTablet for tablet dcbf9c4cc00c4c368797e63620632176 (DEFAULT_TABLE table=test-table [id=ba3483bf2a9e487d8ac7df29d2066259]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:43.057986 6279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbf9c4cc00c4c368797e63620632176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:43.058178 5971 tablet_service.cc:1467] Processing CreateTablet for tablet dcbf9c4cc00c4c368797e63620632176 (DEFAULT_TABLE table=test-table [id=ba3483bf2a9e487d8ac7df29d2066259]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:43.058480 5971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcbf9c4cc00c4c368797e63620632176. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:43.061652 6401 tablet_bootstrap.cc:492] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: Bootstrap starting.
I20260810 23:05:43.062381 6401 tablet_bootstrap.cc:654] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:43.062770 6401 log.cc:824] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:43.063213 6402 tablet_bootstrap.cc:492] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275: Bootstrap starting.
I20260810 23:05:43.063820 6402 tablet_bootstrap.cc:654] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:43.064049 6402 log.cc:824] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:43.064579 6401 tablet_bootstrap.cc:492] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: No bootstrap required, opened a new log
I20260810 23:05:43.064634 6401 ts_tablet_manager.cc:1397] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:05:43.066061 6400 tablet_bootstrap.cc:492] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: Bootstrap starting.
I20260810 23:05:43.066680 6400 tablet_bootstrap.cc:654] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:43.066777 6401 raft_consensus.cc:348] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } }
I20260810 23:05:43.066939 6401 raft_consensus.cc:374] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:43.066970 6401 raft_consensus.cc:729] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35a617764aca43e3b9daae1f8d3b3d61, State: Initialized, Role: FOLLOWER
I20260810 23:05:43.066958 6400 log.cc:824] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:43.067098 6401 consensus_queue.cc:260] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } }
I20260810 23:05:43.067400 6401 ts_tablet_manager.cc:1428] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:05:43.067507 6402 tablet_bootstrap.cc:492] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275: No bootstrap required, opened a new log
I20260810 23:05:43.067514 6358 heartbeater.cc:503] Master 127.23.56.126:33761 was elected leader, sending a full tablet report...
I20260810 23:05:43.067562 6402 ts_tablet_manager.cc:1397] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260810 23:05:43.069445 6400 tablet_bootstrap.cc:492] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: No bootstrap required, opened a new log
I20260810 23:05:43.069507 6400 ts_tablet_manager.cc:1397] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260810 23:05:43.070233 6402 raft_consensus.cc:348] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } }
I20260810 23:05:43.070377 6402 raft_consensus.cc:374] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:43.070405 6402 raft_consensus.cc:729] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3dd59a2c70a416ea5b44d8c986d3275, State: Initialized, Role: FOLLOWER
I20260810 23:05:43.070523 6402 consensus_queue.cc:260] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } }
I20260810 23:05:43.070763 6402 ts_tablet_manager.cc:1428] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 23:05:43.070865 6047 heartbeater.cc:503] Master 127.23.56.126:33761 was elected leader, sending a full tablet report...
I20260810 23:05:43.071899 6400 raft_consensus.cc:348] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } }
I20260810 23:05:43.072057 6400 raft_consensus.cc:374] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:43.072103 6400 raft_consensus.cc:729] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40dd6044f2034ea38b2ffa7190dfbda5, State: Initialized, Role: FOLLOWER
I20260810 23:05:43.072285 6400 consensus_queue.cc:260] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } }
I20260810 23:05:43.072577 6202 heartbeater.cc:503] Master 127.23.56.126:33761 was elected leader, sending a full tablet report...
I20260810 23:05:43.072791 6400 ts_tablet_manager.cc:1428] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260810 23:05:43.111655 6048 tablet.cc:2369] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:05:43.147929 6409 raft_consensus.cc:482] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:43.148102 6409 raft_consensus.cc:504] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } }
I20260810 23:05:43.148528 6409 leader_election.cc:288] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3dd59a2c70a416ea5b44d8c986d3275 (127.23.56.65:37397), 35a617764aca43e3b9daae1f8d3b3d61 (127.23.56.67:39083)
I20260810 23:05:43.152871 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.149198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:50625 (local address 127.23.56.67:39083)
0810 23:05:43.149247 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:43.149250 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:43.149380 (+ 130us) server_negotiation.cc:407] Connection header received
0810 23:05:43.149482 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:43.149483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:43.149492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:43.149521 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:43.149742 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.150513 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.151891 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.152448 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.152499 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.152508 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:43.152732 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 23:05:43.152775 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 23:05:43.153164 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.148829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:34013 (local address 127.23.56.65:37397)
0810 23:05:43.148877 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:43.148880 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:43.149039 (+ 159us) server_negotiation.cc:407] Connection header received
0810 23:05:43.149146 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:43.149147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:43.149158 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:43.149187 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:43.149410 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.150293 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.152188 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.152745 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.152813 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.152822 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:43.153043 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 23:05:43.153096 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:05:43.153326 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.148994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:39083 (local address 127.23.56.66:50625)
0810 23:05:43.149302 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:43.149312 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:43.149408 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:43.149585 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:43.149586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:43.149587 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:43.149657 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.149658 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.150618 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.150619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.151780 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.151783 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.152586 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.152587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.152650 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.152653 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:05:43.153183 (+ 530us) client_negotiation.cc:240] Negotiation successful
0810 23:05:43.153226 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":77,"threads_started":1}
I20260810 23:05:43.153604 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.148731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:37397 (local address 127.23.56.66:34013)
0810 23:05:43.148959 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:43.148971 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:43.149068 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:43.149253 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:43.149254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:43.149256 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:43.149337 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.149339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.150394 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.150394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.151647 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:43.151650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.152885 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.152885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:43.152964 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.152966 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:05:43.153477 (+ 511us) client_negotiation.cc:240] Negotiation successful
0810 23:05:43.153522 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":34,"thread_start_us":133,"threads_started":1}
I20260810 23:05:43.153738 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbf9c4cc00c4c368797e63620632176" candidate_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a617764aca43e3b9daae1f8d3b3d61" is_pre_election: true
I20260810 23:05:43.153909 6298 raft_consensus.cc:2454] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40dd6044f2034ea38b2ffa7190dfbda5 in term 0.
I20260810 23:05:43.154062 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbf9c4cc00c4c368797e63620632176" candidate_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" is_pre_election: true
I20260810 23:05:43.154181 6070 leader_election.cc:302] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a617764aca43e3b9daae1f8d3b3d61, 40dd6044f2034ea38b2ffa7190dfbda5; no voters:
I20260810 23:05:43.154204 5991 raft_consensus.cc:2454] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40dd6044f2034ea38b2ffa7190dfbda5 in term 0.
I20260810 23:05:43.154320 6409 raft_consensus.cc:2781] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:43.154376 6409 raft_consensus.cc:482] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:43.154398 6409 raft_consensus.cc:3037] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:43.155386 6409 raft_consensus.cc:504] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } }
I20260810 23:05:43.155517 6409 leader_election.cc:288] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [CANDIDATE]: Term 1 election: Requested vote from peers e3dd59a2c70a416ea5b44d8c986d3275 (127.23.56.65:37397), 35a617764aca43e3b9daae1f8d3b3d61 (127.23.56.67:39083)
I20260810 23:05:43.155689 5991 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbf9c4cc00c4c368797e63620632176" candidate_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3dd59a2c70a416ea5b44d8c986d3275"
I20260810 23:05:43.155759 5991 raft_consensus.cc:3037] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:43.155763 6298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcbf9c4cc00c4c368797e63620632176" candidate_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35a617764aca43e3b9daae1f8d3b3d61"
I20260810 23:05:43.155838 6298 raft_consensus.cc:3037] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:43.156651 5991 raft_consensus.cc:2454] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40dd6044f2034ea38b2ffa7190dfbda5 in term 1.
I20260810 23:05:43.156760 6298 raft_consensus.cc:2454] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40dd6044f2034ea38b2ffa7190dfbda5 in term 1.
I20260810 23:05:43.156931 6070 leader_election.cc:302] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35a617764aca43e3b9daae1f8d3b3d61, 40dd6044f2034ea38b2ffa7190dfbda5; no voters:
I20260810 23:05:43.157061 6409 raft_consensus.cc:2781] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:43.157222 6409 raft_consensus.cc:686] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 1 LEADER]: Becoming Leader. State: Replica: 40dd6044f2034ea38b2ffa7190dfbda5, State: Running, Role: LEADER
I20260810 23:05:43.157313 6409 consensus_queue.cc:237] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } }
I20260810 23:05:43.158097 5821 catalog_manager.cc:5495] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40dd6044f2034ea38b2ffa7190dfbda5 (127.23.56.66). New cstate: current_term: 1 leader_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40dd6044f2034ea38b2ffa7190dfbda5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 46741 } health_report { overall_health: HEALTHY } } }
I20260810 23:05:43.181188 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.177720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59040 (local address 127.23.56.66:46741)
0810 23:05:43.177770 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:43.177772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:43.177784 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:05:43.177821 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:43.177821 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:43.177869 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:43.177995 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:43.178158 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.178904 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.180081 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:43.180389 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:43.180460 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:43.180656 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:43.180784 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:43.180861 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:43.181089 (+ 228us) server_negotiation.cc:299] Negotiation successful
0810 23:05:43.181111 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:05:43.184317 5991 raft_consensus.cc:1261] T dcbf9c4cc00c4c368797e63620632176 P e3dd59a2c70a416ea5b44d8c986d3275 [term 1 FOLLOWER]: Refusing update from remote peer 40dd6044f2034ea38b2ffa7190dfbda5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:43.184314 6298 raft_consensus.cc:1261] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [term 1 FOLLOWER]: Refusing update from remote peer 40dd6044f2034ea38b2ffa7190dfbda5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:43.184890 6412 consensus_queue.cc:1035] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35a617764aca43e3b9daae1f8d3b3d61" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 39083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:43.185040 6412 consensus_queue.cc:1035] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3dd59a2c70a416ea5b44d8c986d3275" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:43.192713 6415 mvcc.cc:205] Tried to move back new op lower bound from 7317107274479390720 to 7317107274372870144. Current Snapshot: MvccSnapshot[applied={T|T < 7317107274479390720}]
I20260810 23:05:43.192981 6422 mvcc.cc:205] Tried to move back new op lower bound from 7317107274479390720 to 7317107274372870144. Current Snapshot: MvccSnapshot[applied={T|T < 7317107274479390720}]
I20260810 23:05:43.194432 6416 mvcc.cc:205] Tried to move back new op lower bound from 7317107274479390720 to 7317107274372870144. Current Snapshot: MvccSnapshot[applied={T|T < 7317107274479390720 or (T in {7317107274479390720})}]
I20260810 23:05:43.197252 5821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:43.197359 5821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:43.198148 5821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 571b21befd3346d9bbab30a84b1372d6: Sending DeleteTablet for 3 replicas of tablet dcbf9c4cc00c4c368797e63620632176
I20260810 23:05:43.198697 5971 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbf9c4cc00c4c368797e63620632176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:43 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:42532
I20260810 23:05:43.198946 6279 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbf9c4cc00c4c368797e63620632176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:43 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:38762
I20260810 23:05:43.199242 6430 tablet_replica.cc:330] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: stopping tablet replica
I20260810 23:05:43.199420 6430 raft_consensus.cc:2229] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:43.199411 6123 tablet_service.cc:1514] Processing DeleteTablet for tablet dcbf9c4cc00c4c368797e63620632176 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:43 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:59024
I20260810 23:05:43.199608 6430 raft_consensus.cc:2258] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:43.199879 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 5865
I20260810 23:05:43.200127 6430 ts_tablet_manager.cc:1905] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:43.202173 6430 ts_tablet_manager.cc:1918] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:43.202272 6430 log.cc:1196] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/dcbf9c4cc00c4c368797e63620632176
I20260810 23:05:43.202594 6430 ts_tablet_manager.cc:1939] T dcbf9c4cc00c4c368797e63620632176 P 35a617764aca43e3b9daae1f8d3b3d61: Deleting consensus metadata
I20260810 23:05:43.204367 5803 catalog_manager.cc:4841] TS 35a617764aca43e3b9daae1f8d3b3d61 (127.23.56.67:39083): tablet dcbf9c4cc00c4c368797e63620632176 (table test-table [id=ba3483bf2a9e487d8ac7df29d2066259]) successfully deleted
I20260810 23:05:43.207324 6431 tablet_replica.cc:330] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: stopping tablet replica
I20260810 23:05:43.207438 6431 raft_consensus.cc:2229] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:05:43.207552 6431 raft_consensus.cc:2258] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:43.208364 6431 ts_tablet_manager.cc:1905] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:43.210186 6431 ts_tablet_manager.cc:1918] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:43.210318 6431 log.cc:1196] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/dcbf9c4cc00c4c368797e63620632176
I20260810 23:05:43.210672 6431 ts_tablet_manager.cc:1939] T dcbf9c4cc00c4c368797e63620632176 P 40dd6044f2034ea38b2ffa7190dfbda5: Deleting consensus metadata
I20260810 23:05:43.211194 5804 catalog_manager.cc:4841] TS 40dd6044f2034ea38b2ffa7190dfbda5 (127.23.56.66:46741): tablet dcbf9c4cc00c4c368797e63620632176 (table test-table [id=ba3483bf2a9e487d8ac7df29d2066259]) successfully deleted
W20260810 23:05:43.213853 5803 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:37397 (error 108)
I20260810 23:05:43.214874 6395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:43.214735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:37397 (local address 127.0.0.1:42536)
0810 23:05:43.214789 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:43.214839 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:37397: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 23:05:43.215111 5803 catalog_manager.cc:4584] TS e3dd59a2c70a416ea5b44d8c986d3275 (127.23.56.65:37397): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dcbf9c4cc00c4c368797e63620632176: Network error: Client connection negotiation failed: client connection to 127.23.56.65:37397: connect: Connection refused (error 111)
I20260810 23:05:43.215451 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 6049
Aug 10 23:05:43 dist-test-slave-1xrh krb5kdc[4682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403142, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for HTTP/127.23.56.84@KRBTEST.COM
I20260810 23:05:43.238343 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 6205
I20260810 23:05:43.248454 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 5779
I20260810 23:05:43.272658 23777 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 23:05:43.617409 23777 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 23:05:43.617522 23777 mini_ranger.cc:66] Stopping Ranger...
I20260810 23:05:43.962229 23777 mini_ranger.cc:68] Stopped Ranger
2026-08-10 23:05:43.962 UTC [4718] LOG: received smart shutdown request
2026-08-10 23:05:43.967 UTC [4718] LOG: background worker "logical replication launcher" (PID 4728) exited with exit code 1
2026-08-10 23:05:43.967 UTC [4722] LOG: shutting down
2026-08-10 23:05:44.145 UTC [4718] LOG: database system is shut down
2026-08-10T23:05:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47357 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:44 dist-test-slave-1xrh krb5kdc[6444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:44 dist-test-slave-1xrh krb5kdc[6444](info): set up 2 sockets
Aug 10 23:05:44 dist-test-slave-1xrh krb5kdc[6444](info): commencing operation
krb5kdc: starting...
W20260810 23:05:45.372993 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:05:45.494078 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:41201
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:41201,127.23.56.125:40863,127.23.56.124:35267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:45.575181 6461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:45.575369 6461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:45.575390 6461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:45.576411 6461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:45.576442 6461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:45.576457 6461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:45.576469 6461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:45.576480 6461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:45.577945 6461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:41201,127.23.56.125:40863,127.23.56.124:35267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:41201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:45.578127 6461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:45.578287 6461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:45.582983 6467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:45.582953 6470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:45.582974 6468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:45.583202 6461 server_base.cc:1034] running on GCE node
I20260810 23:05:45.583436 6461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:45.583665 6461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:45.584833 6461 hybrid_clock.cc:648] HybridClock initialized: now 1786403145584809 us; error 42 us; skew 500 ppm
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:45.587335 6461 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:05:45.587796 6461 webserver.cc:460] Webserver started at http://127.23.56.126:41867/ using document root <none> and password file <none>
I20260810 23:05:45.588048 6461 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:45.588115 6461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:45.588263 6461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:45.589275 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "8b05fd45b97f44e78002b90bd85eb986"
format_stamp: "Formatted at 2026-08-10 23:05:45 on dist-test-slave-1xrh"
server_key: "5f01699372746d905d06b28ca3afef06"
server_key_iv: "fc410cea989b982bed9613f4787c2a0f"
server_key_version: "encryptionkey@0"
I20260810 23:05:45.589634 6461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "8b05fd45b97f44e78002b90bd85eb986"
format_stamp: "Formatted at 2026-08-10 23:05:45 on dist-test-slave-1xrh"
server_key: "5f01699372746d905d06b28ca3afef06"
server_key_iv: "fc410cea989b982bed9613f4787c2a0f"
server_key_version: "encryptionkey@0"
I20260810 23:05:45.590920 6461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 23:05:45.591737 6478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:45.591989 6461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 23:05:45.592078 6461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "8b05fd45b97f44e78002b90bd85eb986"
format_stamp: "Formatted at 2026-08-10 23:05:45 on dist-test-slave-1xrh"
server_key: "5f01699372746d905d06b28ca3afef06"
server_key_iv: "fc410cea989b982bed9613f4787c2a0f"
server_key_version: "encryptionkey@0"
I20260810 23:05:45.592159 6461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:45.602895 6461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:45.605144 6461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:45.605302 6461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:45.609548 6461 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:41201
I20260810 23:05:45.609591 6540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:41201 every 8 connection(s)
I20260810 23:05:45.609944 6461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:45.610908 6541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:45.612556 6541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:05:45.618569 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 6461
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:45.618702 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:05:45.618948 23777 external_mini_cluster.cc:1428] Setting key 752b43b9585e47ba772c98a68985c52c
I20260810 23:05:45.622577 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.613244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:41201 (local address 127.0.0.1:58396)
0810 23:05:45.613405 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.613441 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:45.613969 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:45.614206 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:45.614209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:45.614462 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:45.614626 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.614630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.615699 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.615701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.616418 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.616420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.616898 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.616899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.616968 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.617412 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:45.617421 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:45.619156 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:45.621065 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.621073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.621075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.621300 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.621597 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.621598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.621598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.621699 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.622142 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:45.622148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:45.622205 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 23:05:45.622291 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:05:45.622419 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260810 23:05:45.622684 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.613390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58396 (local address 127.23.56.126:41201)
0810 23:05:45.613538 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.613541 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.613568 (+ 27us) server_negotiation.cc:407] Connection header received
0810 23:05:45.614104 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.614106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.614122 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.614189 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.614707 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.615593 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.616504 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.616798 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.616873 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.619251 (+ 2378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.619258 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.619259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.619264 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.620974 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.621383 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.621386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.621387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.621505 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.621777 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.621779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.621779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.622063 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.622146 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.622352 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.622510 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260810 23:05:45.623384 6541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:05:45.623739 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.623604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:44472)
0810 23:05:45.623656 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.623706 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 23:05:45.623994 6481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
I20260810 23:05:45.624447 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.624335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:44480)
0810 23:05:45.624388 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.624418 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 23:05:45.624581 6541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.125:40863: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.23.56.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab.
Entry for principal kudu/127.23.56.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab.
I20260810 23:05:45.646034 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.125:40863
--webserver_interface=127.23.56.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab
--principal=kudu/127.23.56.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:41201,127.23.56.125:40863,127.23.56.124:35267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:45.673815 6541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } attempt: 1
I20260810 23:05:45.674327 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.674185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:44484)
0810 23:05:45.674248 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.674293 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:05:45.674711 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.674620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:44498)
0810 23:05:45.674665 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.674692 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 23:05:45.674810 6541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.125:40863: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
W20260810 23:05:45.726310 6548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:45.726498 6548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:45.726521 6548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:45.727594 6548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:45.727630 6548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:45.727644 6548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:45.727656 6548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:45.727669 6548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:45.729064 6548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:41201,127.23.56.125:40863,127.23.56.124:35267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab
--principal=kudu/127.23.56.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.125:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.56.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:45.729252 6548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:45.729413 6548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:45.734117 6554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:45.734124 6555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:45.734386 6548 server_base.cc:1034] running on GCE node
W20260810 23:05:45.734139 6557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:45.734688 6548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:45.734937 6548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:45.736083 6548 hybrid_clock.cc:648] HybridClock initialized: now 1786403145736059 us; error 40 us; skew 500 ppm
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:45.738511 6548 init.cc:377] Logged in from keytab as kudu/127.23.56.125@KRBTEST.COM (short username kudu)
I20260810 23:05:45.738932 6548 webserver.cc:460] Webserver started at http://127.23.56.125:44123/ using document root <none> and password file <none>
I20260810 23:05:45.739213 6548 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:45.739262 6548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:45.739413 6548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:45.740513 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/data/instance:
uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5"
format_stamp: "Formatted at 2026-08-10 23:05:45 on dist-test-slave-1xrh"
server_key: "364a8cedd2a97eb67eae81c51d7bdfa2"
server_key_iv: "bdadf3bca643b1a87d48364d1c85024e"
server_key_version: "encryptionkey@0"
I20260810 23:05:45.740873 6548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/wal/instance:
uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5"
format_stamp: "Formatted at 2026-08-10 23:05:45 on dist-test-slave-1xrh"
server_key: "364a8cedd2a97eb67eae81c51d7bdfa2"
server_key_iv: "bdadf3bca643b1a87d48364d1c85024e"
server_key_version: "encryptionkey@0"
I20260810 23:05:45.742189 6548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:05:45.742946 6565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:45.742950 6541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } attempt: 2
I20260810 23:05:45.743206 6548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:05:45.743309 6548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/wal
uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5"
format_stamp: "Formatted at 2026-08-10 23:05:45 on dist-test-slave-1xrh"
server_key: "364a8cedd2a97eb67eae81c51d7bdfa2"
server_key_iv: "bdadf3bca643b1a87d48364d1c85024e"
server_key_version: "encryptionkey@0"
I20260810 23:05:45.743356 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.743216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:44504)
0810 23:05:45.743278 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.743316 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:05:45.743384 6548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:45.743716 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.743638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:44510)
0810 23:05:45.743669 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.743699 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260810 23:05:45.743834 6541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.125:40863: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
I20260810 23:05:45.758873 6548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:45.760725 6548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:45.760896 6548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:45.764937 6548 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.125:40863
I20260810 23:05:45.764971 6626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.125:40863 every 8 connection(s)
I20260810 23:05:45.765323 6548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
I20260810 23:05:45.766177 6627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:45.767733 6627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:05:45.770747 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 6548
I20260810 23:05:45.770916 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-1/wal/instance
I20260810 23:05:45.771163 23777 external_mini_cluster.cc:1428] Setting key 1c60a6c7f883549c5484abef3751f588
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:45.776970 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.768414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58412 (local address 127.23.56.126:41201)
0810 23:05:45.768493 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.768495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.768683 (+ 188us) server_negotiation.cc:407] Connection header received
0810 23:05:45.769171 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.769173 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.769190 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.769255 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.769850 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.770334 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.771490 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.771693 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.771784 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.774242 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.774243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.774243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.774249 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.775631 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.776037 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.776038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.776039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.776108 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.776347 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.776348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.776348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.776460 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.776542 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.776799 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.776885 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:05:45.777143 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.768402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:41201 (local address 127.0.0.1:58412)
0810 23:05:45.768571 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.768605 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:45.769067 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:45.769328 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:45.769331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:45.769574 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:45.769755 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.769759 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.770431 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.770432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.771401 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.771404 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.771807 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.771808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.771937 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.772497 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:45.772515 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:45.774152 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:45.775718 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.775725 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.775728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.775951 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.776180 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.776181 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.776181 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.776285 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.776535 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:45.776541 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:45.776672 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 23:05:45.776767 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 23:05:45.776916 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260810 23:05:45.777679 6627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.56.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.124@KRBTEST.COM" created.
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
I20260810 23:05:45.786298 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.777917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:44520)
0810 23:05:45.777963 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.777972 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:45.778010 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:45.778281 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:45.778282 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:45.778416 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:45.778458 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.778459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.779373 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.779374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.779850 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.779852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.780372 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.780373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.780436 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.780782 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:45.780786 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:45.783401 (+ 2615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:45.785238 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.785239 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.785240 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.785397 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.785612 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.785613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.785614 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.785666 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.785971 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:45.785972 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:45.786009 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:05:45.786075 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 23:05:45.786170 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 23:05:45.786387 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.778013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44520 (local address 127.23.56.125:40863)
0810 23:05:45.778141 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.778146 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.778162 (+ 16us) server_negotiation.cc:407] Connection header received
0810 23:05:45.778180 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.778182 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.778200 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.778280 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.778532 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.779278 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.779926 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.780284 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.780371 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.783483 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.783492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.783493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.783500 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.785143 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.785490 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.785493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.785494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.785548 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.785738 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.785739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.785740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.785900 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.786001 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.786103 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.786242 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":103,"thread_start_us":80,"threads_started":1}
I20260810 23:05:45.786880 6627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:05:45.787238 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.787101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:45998)
0810 23:05:45.787167 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.787210 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 23:05:45.787395 6566 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
I20260810 23:05:45.787890 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.787802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46002)
0810 23:05:45.787851 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.787875 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 23:05:45.788017 6627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:35267: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab.
Entry for principal kudu/127.23.56.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab.
I20260810 23:05:45.800160 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.124:35267
--webserver_interface=127.23.56.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab
--principal=kudu/127.23.56.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:41201,127.23.56.125:40863,127.23.56.124:35267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:45.835008 6541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } attempt: 3
I20260810 23:05:45.837245 6627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } attempt: 1
I20260810 23:05:45.837647 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.837528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46014)
0810 23:05:45.837586 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.837615 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:05:45.838011 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.837893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46026)
0810 23:05:45.837946 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.837977 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:05:45.838160 6627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:35267: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
I20260810 23:05:45.841914 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.835355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:44524)
0810 23:05:45.835421 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.835432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:45.835501 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:45.835649 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:45.835650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:45.835812 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:45.835877 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.835878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.836556 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.836557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.837084 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.837086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.837557 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.837558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.837642 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.838161 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:45.838165 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:45.839413 (+ 1248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:45.841032 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.841033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.841033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.841194 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.841395 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.841396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.841396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.841444 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.841695 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:45.841695 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:45.841732 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:05:45.841782 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 23:05:45.841854 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 23:05:45.841980 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.835434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.23.56.125:40863)
0810 23:05:45.835510 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.835512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.835523 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:05:45.835555 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.835556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.835567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.835641 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.835972 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.836460 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.837202 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.837469 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.837548 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.839498 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.839500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.839500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.839508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.840955 (+ 1447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.841264 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.841265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.841266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.841332 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.841502 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.841503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.841503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.841634 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.841699 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.841827 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.841902 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:05:45.842386 6541 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:05:45.842713 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.842618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46030)
0810 23:05:45.842668 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.842694 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:05:45.843025 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.842938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46042)
0810 23:05:45.842983 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.843006 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 23:05:45.843142 6541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:35267: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
I20260810 23:05:45.874269 6541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } attempt: 1
I20260810 23:05:45.874711 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.874589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46044)
0810 23:05:45.874648 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.874685 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 23:05:45.875080 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.875000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46056)
0810 23:05:45.875041 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.875065 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 23:05:45.875187 6541 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:35267: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
W20260810 23:05:45.884342 6634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:45.884582 6634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:45.884610 6634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:45.885690 6634 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:45.885723 6634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:45.885738 6634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:45.885751 6634 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:45.885762 6634 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:45.887283 6634 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:41201,127.23.56.125:40863,127.23.56.124:35267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab
--principal=kudu/127.23.56.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.124:35267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.56.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:45.887461 6634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:45.887642 6634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:45.892397 6639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:45.892452 6640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:45.892494 6634 server_base.cc:1034] running on GCE node
W20260810 23:05:45.892468 6642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:45.892835 6634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:45.893077 6634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:45.894228 6634 hybrid_clock.cc:648] HybridClock initialized: now 1786403145894207 us; error 38 us; skew 500 ppm
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:45.896750 6634 init.cc:377] Logged in from keytab as kudu/127.23.56.124@KRBTEST.COM (short username kudu)
I20260810 23:05:45.897166 6634 webserver.cc:460] Webserver started at http://127.23.56.124:36183/ using document root <none> and password file <none>
I20260810 23:05:45.897393 6634 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:45.897465 6634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:45.897648 6634 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:45.898644 6634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/data/instance:
uuid: "e8b82fb8ac31431eae1ee413cd7f79eb"
format_stamp: "Formatted at 2026-08-10 23:05:45 on dist-test-slave-1xrh"
server_key: "24650a65480fddf897d86c76ce6d643c"
server_key_iv: "4b542decd107c112babd7052fa4d545c"
server_key_version: "encryptionkey@0"
I20260810 23:05:45.898991 6634 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/wal/instance:
uuid: "e8b82fb8ac31431eae1ee413cd7f79eb"
format_stamp: "Formatted at 2026-08-10 23:05:45 on dist-test-slave-1xrh"
server_key: "24650a65480fddf897d86c76ce6d643c"
server_key_iv: "4b542decd107c112babd7052fa4d545c"
server_key_version: "encryptionkey@0"
I20260810 23:05:45.900378 6634 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:05:45.901137 6650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:45.901326 6634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:05:45.901422 6634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/wal
uuid: "e8b82fb8ac31431eae1ee413cd7f79eb"
format_stamp: "Formatted at 2026-08-10 23:05:45 on dist-test-slave-1xrh"
server_key: "24650a65480fddf897d86c76ce6d643c"
server_key_iv: "4b542decd107c112babd7052fa4d545c"
server_key_version: "encryptionkey@0"
I20260810 23:05:45.901513 6634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:45.906314 6627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } attempt: 2
I20260810 23:05:45.906715 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.906583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46072)
0810 23:05:45.906643 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.906680 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:05:45.907045 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.906965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46086)
0810 23:05:45.907006 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.907029 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260810 23:05:45.907167 6627 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:35267: Network error: Client connection negotiation failed: client connection to 127.23.56.124:35267: connect: Connection refused (error 111)
I20260810 23:05:45.918047 6634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:45.919948 6634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:45.920085 6634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:45.924073 6634 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.124:35267
I20260810 23:05:45.924157 6713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.124:35267 every 8 connection(s)
I20260810 23:05:45.924461 6634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
I20260810 23:05:45.925107 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 6634
I20260810 23:05:45.925225 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786402992661514-23777-0/minicluster-data/master-2/wal/instance
I20260810 23:05:45.925403 23777 external_mini_cluster.cc:1428] Setting key 0e4f204f6225f7d2bdf2465ce4474e16
I20260810 23:05:45.925519 6714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:45.928175 6714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:45.936681 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.925901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46102 (local address 127.23.56.124:35267)
0810 23:05:45.926074 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.926086 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.926110 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:05:45.926730 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.926737 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.926781 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.926889 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.927798 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.928758 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.930245 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.930752 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.930870 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.933412 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.933421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.933421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.933438 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.934999 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.935398 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.935401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.935401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.935489 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.935716 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.935717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.935718 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.936018 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.936137 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.936279 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.936450 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260810 23:05:45.936882 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.928795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:41201 (local address 127.0.0.1:58426)
0810 23:05:45.928928 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.928962 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:45.929022 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:45.929177 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:45.929179 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:45.929414 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:45.929475 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.929477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.930554 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.930556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.931182 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.931186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.931599 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.931601 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.931677 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.932131 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:45.932139 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:45.933842 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:45.935736 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.935742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.935744 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.935972 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.936200 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.936201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.936201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.936271 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.936537 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:45.936542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:45.936588 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 23:05:45.936679 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:05:45.936790 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260810 23:05:45.936893 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.928902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58426 (local address 127.23.56.126:41201)
0810 23:05:45.928955 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.928958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.929028 (+ 70us) server_negotiation.cc:407] Connection header received
0810 23:05:45.929063 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.929076 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.929087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.929169 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.929594 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.930406 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.931275 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.931472 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.931590 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.933915 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.933917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.933917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.933923 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.935668 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.936065 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.936066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.936067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.936127 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.936344 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.936345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.936346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.936477 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.936538 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.936694 (+ 156us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.936794 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 23:05:45.937583 6714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:45.944388 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.937922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44530 (local address 127.23.56.125:40863)
0810 23:05:45.937964 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.937966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.937974 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:05:45.938002 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.938002 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.938012 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.938067 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.938315 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.938773 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.939484 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.939711 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.939788 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.941676 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.941678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.941678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.941685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.943282 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.943629 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.943630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.943630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.943682 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.943892 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.943893 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.943893 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.944011 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.944080 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.944233 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.944301 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:05:45.944388 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.937840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:44530)
0810 23:05:45.937894 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.937906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:45.937951 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:45.938071 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:45.938071 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:45.938207 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:45.938234 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.938235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.938856 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.938858 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.939403 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.939405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.939785 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.939786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.939842 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.940388 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:45.940393 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:45.941590 (+ 1197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:45.943375 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.943376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.943376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.943533 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.943756 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.943757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.943757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.943826 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.944083 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:45.944084 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:45.944124 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:05:45.944207 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:05:45.944297 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 23:05:45.944769 6714 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:05:45.945195 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.937540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.23.56.126:41201)
0810 23:05:45.937582 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.937583 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.937595 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:05:45.937655 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.937656 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.937666 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.937724 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.938322 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.938770 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.939504 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.939708 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.939780 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.942300 (+ 2520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.942301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.942301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.942308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.943761 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.944113 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.944114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.944115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.944168 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.944364 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.944365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.944366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.944627 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.944798 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.944986 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.945084 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
I20260810 23:05:45.950348 6541 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } attempt: 2
I20260810 23:05:45.952072 6718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.945011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46106)
0810 23:05:45.945056 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.945065 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:45.945099 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:45.945219 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:45.945219 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:45.945336 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:45.945360 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.945361 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.946158 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.946159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.946823 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.946825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.947273 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.947274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.947342 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.947648 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:45.947653 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:45.949017 (+ 1364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:45.951020 (+ 2003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.951021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.951021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.951250 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.951435 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.951435 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.951436 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.951484 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.951735 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:45.951736 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:45.951773 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 23:05:45.951843 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:05:45.951952 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:05:45.952198 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.945067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.23.56.124:35267)
0810 23:05:45.945099 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.945101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.945108 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:05:45.945150 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.945151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.945159 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.945215 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.945409 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.946103 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.946886 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.947205 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.947276 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.949107 (+ 1831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.949108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.949108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.949114 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.950947 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.951310 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.951311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.951312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.951370 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.951562 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.951563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.951564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.951668 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.951742 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.951879 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.952018 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":17}
I20260810 23:05:45.954072 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.945911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44546 (local address 127.23.56.125:40863)
0810 23:05:45.945964 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.945966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.945989 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:05:45.946008 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.946009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.946017 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.946064 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.946582 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.947257 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.948155 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.948439 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.948508 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.950847 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.950848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.950848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.950857 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.952859 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.953259 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.953260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.953260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.953315 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.953545 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.953546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.953546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.953665 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.953852 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.953895 (+ 43us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.953968 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260810 23:05:45.954149 6714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb: Bootstrap starting.
I20260810 23:05:45.955044 6714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb: Neither blocks nor log segments found. Creating new log.
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
I20260810 23:05:45.955621 6714 log.cc:824] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:45.956269 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:45.956635 6714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb: No bootstrap required, opened a new log
I20260810 23:05:45.958696 6714 raft_consensus.cc:348] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } }
I20260810 23:05:45.958992 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.950623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46120)
0810 23:05:45.950680 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.950690 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:45.950736 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:45.951066 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:45.951067 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:45.951248 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:45.951286 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.951288 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.952773 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.952774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.953406 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.953408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.953915 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.953916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:45.953973 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.954474 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:45.954478 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:45.955921 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:45.957538 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.957539 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.957539 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.957720 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.957997 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:45.957998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:45.957998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:45.958065 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:45.958664 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:45.958665 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:45.958713 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 23:05:45.958792 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:05:45.958885 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 23:05:45.959338 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.950703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46120 (local address 127.23.56.124:35267)
0810 23:05:45.950904 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.950906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.950913 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:05:45.950930 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.950931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.950941 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.951360 (+ 419us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.951403 (+ 43us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.952657 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.953493 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.953823 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.954203 (+ 380us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.956061 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.956062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.956062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.956069 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.957447 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.957849 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.957851 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.957851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.957916 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.958401 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.958402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.958402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.958579 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.958647 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.959108 (+ 461us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.959227 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":129,"threads_started":1}
I20260810 23:05:45.959653 6714 raft_consensus.cc:374] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:45.959728 6714 raft_consensus.cc:729] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8b82fb8ac31431eae1ee413cd7f79eb, State: Initialized, Role: FOLLOWER
I20260810 23:05:45.959913 6714 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } }
I20260810 23:05:45.960259 6714 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:45.961508 6541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986: Bootstrap starting.
I20260810 23:05:45.961992 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:45.962069 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:05:45.962312 6541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986: Neither blocks nor log segments found. Creating new log.
W20260810 23:05:45.962352 6739 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:45.962393 6739 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:45.962597 6541 log.cc:824] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:45.963447 6541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986: No bootstrap required, opened a new log
I20260810 23:05:45.964149 6634 master_runner.cc:186] Error getting master registration for 127.23.56.126:41201: OK, instance_id { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" instance_seqno: 1786403145608343 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:05:45.966352 6634 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:05:45.966620 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.958803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.23.56.126:41201)
0810 23:05:45.958842 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.958844 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.959351 (+ 507us) server_negotiation.cc:407] Connection header received
0810 23:05:45.959449 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.959450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.959461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.959514 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.960254 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.960945 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.961910 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.962181 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.962270 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.963846 (+ 1576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.963847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.963848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.963856 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.965478 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.965863 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.965864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.965864 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.965917 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.966113 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.966114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.966114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.966220 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.966280 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.966470 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.966540 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:05:45.968531 6541 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } }
I20260810 23:05:45.968739 6541 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:45.968840 6541 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b05fd45b97f44e78002b90bd85eb986, State: Initialized, Role: FOLLOWER
I20260810 23:05:45.969025 6541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } }
I20260810 23:05:45.969385 6541 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:45.972203 6631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.959057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44558 (local address 127.23.56.125:40863)
0810 23:05:45.959097 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.959100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.963370 (+ 4270us) server_negotiation.cc:407] Connection header received
0810 23:05:45.963490 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.963492 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.963503 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.963536 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.964454 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.965461 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.966418 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.966696 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.966740 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.968209 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.968210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.968211 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.968217 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.970523 (+ 2306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.971244 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.971245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.971245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.971298 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.971549 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.971550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.971550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.971676 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.971771 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.972001 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.972079 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:05:45.972908 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.958289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.23.56.124:35267)
0810 23:05:45.958331 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.958333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.958458 (+ 125us) server_negotiation.cc:407] Connection header received
0810 23:05:45.958551 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.958552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.958561 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.958590 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.959433 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.960085 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.968161 (+ 8076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.968456 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.968499 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:45.969951 (+ 1452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:45.969952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:45.969953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:45.969960 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:45.971659 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.972064 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.972065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.972065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.972117 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:45.972346 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:45.972347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:45.972347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:45.972481 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:45.972545 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:45.972764 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 23:05:45.972841 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:05:45.971932 6461 master_runner.cc:186] Error getting master registration for 127.23.56.125:40863: OK, instance_id { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" instance_seqno: 1786403145763734 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:05:45.974897 6461 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 23:05:45.975647 6754 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:45.975713 6754 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:45.976070 6742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:45.976151 6742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:05:45.998291 6627 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } attempt: 3
I20260810 23:05:45.999516 6742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:45.999701 6742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } }
I20260810 23:05:46.000044 6742 leader_election.cc:288] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a515d43e14f4cebbe4ef1f77e8ce2e5 (127.23.56.125:40863), e8b82fb8ac31431eae1ee413cd7f79eb (127.23.56.124:35267)
I20260810 23:05:46.000484 6684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b05fd45b97f44e78002b90bd85eb986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" is_pre_election: true
I20260810 23:05:46.000636 6684 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b05fd45b97f44e78002b90bd85eb986 in term 0.
I20260810 23:05:46.000862 6479 leader_election.cc:302] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b05fd45b97f44e78002b90bd85eb986, e8b82fb8ac31431eae1ee413cd7f79eb; no voters:
I20260810 23:05:46.001121 6742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:46.001320 6742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:46.001792 6742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:46.002930 6742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } }
Aug 10 23:05:45 dist-test-slave-1xrh krb5kdc[6444](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403145, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
I20260810 23:05:46.003563 6684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b05fd45b97f44e78002b90bd85eb986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb"
I20260810 23:05:46.003661 6684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:46.005333 6684 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b05fd45b97f44e78002b90bd85eb986 in term 1.
I20260810 23:05:46.005601 6479 leader_election.cc:302] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b05fd45b97f44e78002b90bd85eb986, e8b82fb8ac31431eae1ee413cd7f79eb; no voters:
I20260810 23:05:46.005853 6742 leader_election.cc:288] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [CANDIDATE]: Term 1 election: Requested vote from peers 4a515d43e14f4cebbe4ef1f77e8ce2e5 (127.23.56.125:40863), e8b82fb8ac31431eae1ee413cd7f79eb (127.23.56.124:35267)
I20260810 23:05:46.005996 6742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:46.006554 6742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [term 1 LEADER]: Becoming Leader. State: Replica: 8b05fd45b97f44e78002b90bd85eb986, State: Running, Role: LEADER
I20260810 23:05:46.006764 6742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } }
I20260810 23:05:46.007380 6629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.998597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:35267 (local address 127.0.0.1:46126)
0810 23:05:45.998650 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:45.998663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:45.998702 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:45.998837 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:45.998838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:45.999003 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:45.999047 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:45.999049 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.999924 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:45.999925 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:46.000684 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:46.000686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:46.001197 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:46.001198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:46.001257 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:46.001785 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:46.001792 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:46.003408 (+ 1616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:46.005507 (+ 2099us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:46.005508 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:46.005508 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:46.005762 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:46.006524 (+ 762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:46.006525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:46.006525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:46.006607 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:46.007042 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:46.007043 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:46.007099 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 23:05:46.007185 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:05:46.007288 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260810 23:05:46.007933 6717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:45.998636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46126 (local address 127.23.56.124:35267)
0810 23:05:45.998674 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:45.998676 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:45.998686 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:05:45.998750 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:45.998751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:45.998761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:45.998833 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:45.999140 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:45.999797 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:46.000773 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:46.001067 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:46.001135 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:46.003521 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:46.003522 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:46.003522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:46.003529 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:46.005378 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:46.006268 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:46.006269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:46.006270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:46.006348 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:46.006775 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:46.006776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:46.006777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:46.006936 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:46.007001 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:46.007627 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 23:05:46.007749 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:05:46.008208 6756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b05fd45b97f44e78002b90bd85eb986. Latest consensus state: current_term: 1 leader_uuid: "8b05fd45b97f44e78002b90bd85eb986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:46.008338 6756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:46.008648 6758 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:46.009003 6758 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:46.010020 6627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5: Bootstrap starting.
I20260810 23:05:46.010778 6627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:46.011080 6627 log.cc:824] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:46.012147 6684 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [term 1 FOLLOWER]: Refusing update from remote peer 8b05fd45b97f44e78002b90bd85eb986: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:46.012835 6627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5: No bootstrap required, opened a new log
I20260810 23:05:46.014868 6627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } }
I20260810 23:05:46.014995 6627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:46.015036 6627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a515d43e14f4cebbe4ef1f77e8ce2e5, State: Initialized, Role: FOLLOWER
I20260810 23:05:46.015169 6627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } }
I20260810 23:05:46.015538 6742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:46.015527 6597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b05fd45b97f44e78002b90bd85eb986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" is_pre_election: true
I20260810 23:05:46.015666 6597 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b05fd45b97f44e78002b90bd85eb986 in term 0.
I20260810 23:05:46.015542 6595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8b05fd45b97f44e78002b90bd85eb986" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5"
I20260810 23:05:46.015928 6595 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:46.016752 6595 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b05fd45b97f44e78002b90bd85eb986 in term 1.
I20260810 23:05:46.015506 6627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:46.017020 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b05fd45b97f44e78002b90bd85eb986. Latest consensus state: current_term: 1 leader_uuid: "8b05fd45b97f44e78002b90bd85eb986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:46.017112 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:05:46.017611 6595 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [term 1 FOLLOWER]: Refusing update from remote peer 8b05fd45b97f44e78002b90bd85eb986: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:46.017884 6742 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:46.015486 6762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:46.018805 6762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:05:46.019073 6762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b05fd45b97f44e78002b90bd85eb986. Latest consensus state: current_term: 1 leader_uuid: "8b05fd45b97f44e78002b90bd85eb986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:46.020543 6742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b05fd45b97f44e78002b90bd85eb986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:46.020624 6742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:46.021090 6759 mvcc.cc:205] Tried to move back new op lower bound from 7317107286063620096 to 7317107286045396992. Current Snapshot: MvccSnapshot[applied={T|T < 7317107286063620096}]
I20260810 23:05:46.021175 6762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:05:46.021728 6762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b05fd45b97f44e78002b90bd85eb986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:46.021842 6762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:05:46.021931 6777 mvcc.cc:205] Tried to move back new op lower bound from 7317107286063620096 to 7317107286045396992. Current Snapshot: MvccSnapshot[applied={T|T < 7317107286063620096}]
I20260810 23:05:46.023167 6756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8b05fd45b97f44e78002b90bd85eb986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:46.023249 6756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:46.023640 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8b05fd45b97f44e78002b90bd85eb986" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b05fd45b97f44e78002b90bd85eb986" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 41201 } } peers { permanent_uuid: "4a515d43e14f4cebbe4ef1f77e8ce2e5" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "e8b82fb8ac31431eae1ee413cd7f79eb" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 35267 } } }
I20260810 23:05:46.023730 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:05:46.023723 6780 catalog_manager.cc:1259] Loaded cluster ID: feed2523036c4db6adcd7cc4dfa77a0f
I20260810 23:05:46.023765 6780 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5: loading cluster ID for follower catalog manager: success
W20260810 23:05:46.023882 6780 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260810 23:05:46.023927 6780 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:46.023975 6764 mvcc.cc:205] Tried to move back new op lower bound from 7317107286063620096 to 7317107286045396992. Current Snapshot: MvccSnapshot[applied={T|T < 7317107286063620096}]
I20260810 23:05:46.024334 6758 catalog_manager.cc:1347] Generated new cluster ID: feed2523036c4db6adcd7cc4dfa77a0f
I20260810 23:05:46.024410 6758 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:46.038997 6758 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:46.039841 6758 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:46.046030 6758 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b05fd45b97f44e78002b90bd85eb986: Generated new TSK 0
I20260810 23:05:46.046248 6758 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:46.962849 6739 catalog_manager.cc:1259] Loaded cluster ID: feed2523036c4db6adcd7cc4dfa77a0f
I20260810 23:05:46.962934 6739 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb: loading cluster ID for follower catalog manager: success
I20260810 23:05:46.963737 6739 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb: acquiring CA information for follower catalog manager: success
I20260810 23:05:46.964083 6739 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e8b82fb8ac31431eae1ee413cd7f79eb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:05:47.025020 6780 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5: acquiring CA information for follower catalog manager: success
I20260810 23:05:47.025315 6780 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4a515d43e14f4cebbe4ef1f77e8ce2e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:05:47.105082 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 6461
I20260810 23:05:47.112983 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 6548
I20260810 23:05:47.120189 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 6634
2026-08-10T23:05:47Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2857 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:47 dist-test-slave-1xrh krb5kdc[6796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:47 dist-test-slave-1xrh krb5kdc[6796](info): set up 2 sockets
Aug 10 23:05:47 dist-test-slave-1xrh krb5kdc[6796](info): commencing operation
krb5kdc: starting...
W20260810 23:05:48.227747 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:05:48 dist-test-slave-1xrh krb5kdc[6796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:05:48.351583 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45911
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:40715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:45911
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:48.432840 6816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:48.433043 6816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:48.433064 6816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:48.434104 6816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:48.434139 6816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:48.434154 6816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:48.434165 6816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:48.434178 6816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:48.435545 6816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:40715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:45911
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:48.435722 6816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:48.435882 6816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:48.440599 6823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:48.440589 6825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:48.440653 6816 server_base.cc:1034] running on GCE node
W20260810 23:05:48.440590 6822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:48.441038 6816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:48.441344 6816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:48.442493 6816 hybrid_clock.cc:648] HybridClock initialized: now 1786403148442469 us; error 39 us; skew 500 ppm
Aug 10 23:05:48 dist-test-slave-1xrh krb5kdc[6796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403148, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:48.450387 6816 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:05:48.450853 6816 webserver.cc:460] Webserver started at http://127.23.56.126:39151/ using document root <none> and password file <none>
I20260810 23:05:48.451079 6816 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:48.451167 6816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:48.451324 6816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:48.452410 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "98f37a3742aa4384b706f113cde6f22e"
format_stamp: "Formatted at 2026-08-10 23:05:48 on dist-test-slave-1xrh"
server_key: "92000e8ee487fe6074f7ed181a1c0dcb"
server_key_iv: "4b3b9b91cc8098c40beff5881fc76a04"
server_key_version: "encryptionkey@0"
I20260810 23:05:48.452761 6816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "98f37a3742aa4384b706f113cde6f22e"
format_stamp: "Formatted at 2026-08-10 23:05:48 on dist-test-slave-1xrh"
server_key: "92000e8ee487fe6074f7ed181a1c0dcb"
server_key_iv: "4b3b9b91cc8098c40beff5881fc76a04"
server_key_version: "encryptionkey@0"
I20260810 23:05:48.454044 6816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:05:48.454802 6834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:48.455015 6816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:48.455101 6816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "98f37a3742aa4384b706f113cde6f22e"
format_stamp: "Formatted at 2026-08-10 23:05:48 on dist-test-slave-1xrh"
server_key: "92000e8ee487fe6074f7ed181a1c0dcb"
server_key_iv: "4b3b9b91cc8098c40beff5881fc76a04"
server_key_version: "encryptionkey@0"
I20260810 23:05:48.455212 6816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:48.464824 6816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:48.466646 6816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:48.466753 6816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:48.470507 6816 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:45911
I20260810 23:05:48.470551 6889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:45911 every 8 connection(s)
I20260810 23:05:48.470906 6816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:48.471819 6891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:48.474117 6891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e: Bootstrap starting.
I20260810 23:05:48.474678 6891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:48.474968 6891 log.cc:824] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:48.475668 6891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e: No bootstrap required, opened a new log
I20260810 23:05:48.476598 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 6816
I20260810 23:05:48.476677 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:05:48.476862 23777 external_mini_cluster.cc:1428] Setting key b82a24a4ceadd44a5eddc732303627e1
I20260810 23:05:48.477742 6891 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f37a3742aa4384b706f113cde6f22e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45911 } }
I20260810 23:05:48.477913 6891 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:48.477952 6891 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f37a3742aa4384b706f113cde6f22e, State: Initialized, Role: FOLLOWER
I20260810 23:05:48.478120 6891 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f37a3742aa4384b706f113cde6f22e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45911 } }
I20260810 23:05:48.478204 6891 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:48.478227 6891 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:48.478299 6891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:48.479396 6891 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f37a3742aa4384b706f113cde6f22e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45911 } }
I20260810 23:05:48.479502 6891 leader_election.cc:302] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98f37a3742aa4384b706f113cde6f22e; no voters:
I20260810 23:05:48.479656 6891 leader_election.cc:288] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:48.479710 6897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:48.479887 6897 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [term 1 LEADER]: Becoming Leader. State: Replica: 98f37a3742aa4384b706f113cde6f22e, State: Running, Role: LEADER
I20260810 23:05:48.479889 6891 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:48.480053 6897 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f37a3742aa4384b706f113cde6f22e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45911 } }
I20260810 23:05:48.481498 6898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98f37a3742aa4384b706f113cde6f22e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f37a3742aa4384b706f113cde6f22e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45911 } } }
I20260810 23:05:48.481503 6899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98f37a3742aa4384b706f113cde6f22e. Latest consensus state: current_term: 1 leader_uuid: "98f37a3742aa4384b706f113cde6f22e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f37a3742aa4384b706f113cde6f22e" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45911 } } }
I20260810 23:05:48.481845 6898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:48.481884 6899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e [sys.catalog]: This master's current role is: LEADER
W20260810 23:05:48.482040 6916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:05:48.482110 6916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:05:48.482204 6919 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:48.482363 6919 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:48.483848 6919 catalog_manager.cc:1347] Generated new cluster ID: c4071681d75747e8abe345f90093f14a
I20260810 23:05:48.483901 6919 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:05:48 dist-test-slave-1xrh krb5kdc[6796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:48.488703 6896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:48.477344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.23.56.126:45911)
0810 23:05:48.477560 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:48.477570 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:48.477605 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:05:48.477720 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:48.477727 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:48.477793 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:48.477951 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:48.478698 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:48.480113 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:48.481248 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:48.481884 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:48.482000 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:48.484994 (+ 2994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:48.484998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:48.484999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:48.485010 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:48.486915 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:48.487330 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:48.487333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:48.487334 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:48.487462 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:48.487703 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:48.487704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:48.487705 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:48.488007 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:48.488173 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:48.488246 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 23:05:48.488399 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260810 23:05:48.489643 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:05:48.492589 6919 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:48.493263 6919 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:48.496132 6919 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98f37a3742aa4384b706f113cde6f22e: Generated new TSK 0
I20260810 23:05:48.496284 6919 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:05:48.543949 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 6816
2026-08-10T23:05:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1423 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:05:48 dist-test-slave-1xrh krb5kdc[6924](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:05:48 dist-test-slave-1xrh krb5kdc[6924](info): set up 2 sockets
Aug 10 23:05:48 dist-test-slave-1xrh krb5kdc[6924](info): commencing operation
krb5kdc: starting...
W20260810 23:05:49.640327 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:05:49 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:05:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:05:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:05:49.762332 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38651
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:38651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:49.848397 6939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:49.848608 6939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:49.848632 6939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:49.849869 6939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:49.849911 6939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:49.849925 6939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:49.849978 6939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:49.849996 6939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:49.851456 6939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:38651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:49.851697 6939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:49.851936 6939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:49.856822 6944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:49.856860 6949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:49.856878 6945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:49.856909 6939 server_base.cc:1034] running on GCE node
I20260810 23:05:49.857317 6939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:49.857604 6939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:49.858747 6939 hybrid_clock.cc:648] HybridClock initialized: now 1786403149858726 us; error 40 us; skew 500 ppm
Aug 10 23:05:49 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403149, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:49.861186 6939 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:05:49.861630 6939 webserver.cc:460] Webserver started at http://127.23.56.126:40961/ using document root <none> and password file <none>
I20260810 23:05:49.861846 6939 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:49.861924 6939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:49.862099 6939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:49.863098 6939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "314daad9919d4a7887cb9397cfa6e6f3"
format_stamp: "Formatted at 2026-08-10 23:05:49 on dist-test-slave-1xrh"
server_key: "08531869c8264b2fc97dbae4a875c376"
server_key_iv: "76b5998b7eb99e02aa944601a1a2301d"
server_key_version: "encryptionkey@0"
I20260810 23:05:49.863528 6939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "314daad9919d4a7887cb9397cfa6e6f3"
format_stamp: "Formatted at 2026-08-10 23:05:49 on dist-test-slave-1xrh"
server_key: "08531869c8264b2fc97dbae4a875c376"
server_key_iv: "76b5998b7eb99e02aa944601a1a2301d"
server_key_version: "encryptionkey@0"
I20260810 23:05:49.864892 6939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:05:49.865612 6958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:49.865799 6939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:49.865895 6939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "314daad9919d4a7887cb9397cfa6e6f3"
format_stamp: "Formatted at 2026-08-10 23:05:49 on dist-test-slave-1xrh"
server_key: "08531869c8264b2fc97dbae4a875c376"
server_key_iv: "76b5998b7eb99e02aa944601a1a2301d"
server_key_version: "encryptionkey@0"
I20260810 23:05:49.865969 6939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:49.884307 6939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:49.885991 6939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:49.886147 6939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:49.890100 6939 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:38651
I20260810 23:05:49.890134 7017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:38651 every 8 connection(s)
I20260810 23:05:49.890484 6939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:49.891624 7018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:49.893788 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Bootstrap starting.
I20260810 23:05:49.894366 7018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:49.894649 7018 log.cc:824] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:49.895386 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: No bootstrap required, opened a new log
I20260810 23:05:49.896924 7018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:49.897053 7018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:49.897111 7018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 314daad9919d4a7887cb9397cfa6e6f3, State: Initialized, Role: FOLLOWER
I20260810 23:05:49.897297 7018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:49.897394 7018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:49.897418 7018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:49.897487 7018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:49.897445 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 6939
I20260810 23:05:49.897573 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:05:49.897763 23777 external_mini_cluster.cc:1428] Setting key 22793243e20c6105e35790ce825fe95c
I20260810 23:05:49.898576 7018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:49.898679 7018 leader_election.cc:302] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 314daad9919d4a7887cb9397cfa6e6f3; no voters:
I20260810 23:05:49.898831 7018 leader_election.cc:288] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:05:49.898882 7025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:49.899063 7025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 1 LEADER]: Becoming Leader. State: Replica: 314daad9919d4a7887cb9397cfa6e6f3, State: Running, Role: LEADER
I20260810 23:05:49.899147 7018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:49.899236 7025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:49.899765 7027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 314daad9919d4a7887cb9397cfa6e6f3. Latest consensus state: current_term: 1 leader_uuid: "314daad9919d4a7887cb9397cfa6e6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } } }
I20260810 23:05:49.899883 7027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:49.900096 7034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:49.900662 7034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:49.900837 7026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "314daad9919d4a7887cb9397cfa6e6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } } }
I20260810 23:05:49.901113 7026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:49.902355 7034 catalog_manager.cc:1347] Generated new cluster ID: e44427ddb0f34cb3b45fdd4ce63a66b7
I20260810 23:05:49.902407 7034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:05:49 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403149, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:49.909407 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:49.898272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.23.56.126:38651)
0810 23:05:49.898455 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:49.898465 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:49.898500 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:05:49.898614 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:49.898621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:49.898679 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:49.898852 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:49.899736 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:49.901283 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:49.902189 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:49.902788 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:49.902888 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:49.905673 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:49.905678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:49.905678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:49.905689 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:49.907767 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:49.908142 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:49.908145 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:49.908145 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:49.908298 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:49.908515 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:49.908516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:49.908516 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:49.908841 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:49.909016 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:49.909081 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 23:05:49.909214 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260810 23:05:49.914323 7034 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:05:49.914793 7034 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:49.922693 7034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Generated new TSK 0
I20260810 23:05:49.922915 7034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:05:49.950104 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38651
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:50.032922 7047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:50.033160 7047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:50.033200 7047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:50.034332 7047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:50.034401 7047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:50.034449 7047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:05:50.035983 7047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:50.036235 7047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:50.036465 7047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:50.041388 7054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.041383 7055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.041393 7057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:50.041548 7047 server_base.cc:1034] running on GCE node
I20260810 23:05:50.041802 7047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:50.042019 7047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:50.043179 7047 hybrid_clock.cc:648] HybridClock initialized: now 1786403150043153 us; error 48 us; skew 500 ppm
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:50.045838 7047 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:05:50.046216 7047 webserver.cc:460] Webserver started at http://127.23.56.65:33653/ using document root <none> and password file <none>
I20260810 23:05:50.046409 7047 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:50.046452 7047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:50.046540 7047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:50.047603 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "d079a63756c84a569fe91f0fbeaeadd3"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "c46e2ca90106b71c332cebc92e9a693a"
server_key_iv: "754947cfa6730b5cd125c553633bef4f"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.047909 7047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "d079a63756c84a569fe91f0fbeaeadd3"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "c46e2ca90106b71c332cebc92e9a693a"
server_key_iv: "754947cfa6730b5cd125c553633bef4f"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.049153 7047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:05:50.049866 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.050057 7047 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:05:50.050140 7047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "d079a63756c84a569fe91f0fbeaeadd3"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "c46e2ca90106b71c332cebc92e9a693a"
server_key_iv: "754947cfa6730b5cd125c553633bef4f"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.050194 7047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:50.065725 7047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:50.067627 7047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:50.067736 7047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:50.067907 7047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:50.068339 7047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:50.068385 7047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.068409 7047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:50.068457 7047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.074694 7047 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:35609
I20260810 23:05:50.074729 7200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:35609 every 8 connection(s)
I20260810 23:05:50.075079 7047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:50.075484 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 7047
I20260810 23:05:50.075584 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:05:50.075795 23777 external_mini_cluster.cc:1428] Setting key ee4406832b2c9d361906c1e304b04310
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
I20260810 23:05:50.085866 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.075899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:45917 (local address 127.23.56.126:38651)
0810 23:05:50.075949 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.075952 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.076258 (+ 306us) server_negotiation.cc:407] Connection header received
0810 23:05:50.076882 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.076884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.076896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.076951 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:50.077916 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.078727 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.079805 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.080147 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.080225 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.082620 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:50.082622 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:50.082622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:50.082631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:50.084393 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.084780 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.084782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.084782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.084852 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.085101 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.085102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.085103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.085283 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:50.085358 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.085623 (+ 265us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.085730 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:05:50.086007 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.075918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:45917)
0810 23:05:50.076110 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.076161 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.076775 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.077025 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.077029 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.077611 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:50.077807 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.077813 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.078841 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.078843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.079696 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.079699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.080237 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.080238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.080352 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.080901 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:50.080912 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:50.082525 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:50.084477 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.084484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.084486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.084698 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.084929 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.084930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.084930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.085029 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.085365 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:50.085370 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:50.085479 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.085597 (+ 118us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.085754 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260810 23:05:50.086583 7201 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38651
I20260810 23:05:50.086714 7201 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:50.086977 7201 heartbeater.cc:511] Master 127.23.56.126:38651 requested a full tablet report, sending...
I20260810 23:05:50.087612 6978 ts_manager.cc:194] Registered new tserver with Master: d079a63756c84a569fe91f0fbeaeadd3 (127.23.56.65:35609)
I20260810 23:05:50.088370 6978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:45917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:05:50.104127 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38651
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:50.185393 7206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:50.185597 7206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:50.185621 7206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:50.186764 7206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:50.186805 7206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:50.186829 7206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:05:50.188258 7206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:50.188488 7206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:50.188727 7206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:50.193688 7216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.193662 7213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:50.193858 7206 server_base.cc:1034] running on GCE node
W20260810 23:05:50.193720 7214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:50.194126 7206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:50.194386 7206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:50.195533 7206 hybrid_clock.cc:648] HybridClock initialized: now 1786403150195513 us; error 35 us; skew 500 ppm
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:50.198038 7206 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:05:50.198531 7206 webserver.cc:460] Webserver started at http://127.23.56.66:41609/ using document root <none> and password file <none>
I20260810 23:05:50.198763 7206 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:50.198807 7206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:50.198949 7206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:50.200150 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "32a6dc92f3634db384160aa638dc3f9d"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "a7173a3e7dea2bc38b8372b7bc5e6033"
server_key_iv: "db2269be9998f160b0238ab056ea54d6"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.200572 7206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "32a6dc92f3634db384160aa638dc3f9d"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "a7173a3e7dea2bc38b8372b7bc5e6033"
server_key_iv: "db2269be9998f160b0238ab056ea54d6"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.201870 7206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:05:50.202557 7224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.202769 7206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:05:50.202850 7206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "32a6dc92f3634db384160aa638dc3f9d"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "a7173a3e7dea2bc38b8372b7bc5e6033"
server_key_iv: "db2269be9998f160b0238ab056ea54d6"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.202893 7206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:50.216800 7206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:50.218783 7206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:50.218938 7206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:50.219214 7206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:50.219527 7206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:50.219580 7206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.219612 7206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:50.219640 7206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.225736 7206 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:40009
I20260810 23:05:50.225787 7356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:40009 every 8 connection(s)
I20260810 23:05:50.226118 7206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:50.229312 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 7206
I20260810 23:05:50.229414 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:05:50.229609 23777 external_mini_cluster.cc:1428] Setting key 8d3d101457c001e9a1a9589d96744a19
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:50.236243 7360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.226737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:43801)
0810 23:05:50.226890 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.226952 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.227481 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.227692 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.227696 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.228268 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:50.228461 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.228465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.229167 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.229169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.230438 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.230441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.230806 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.230808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.230953 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.231489 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:50.231500 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:50.233082 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:50.234882 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.234888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.234890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.235100 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.235380 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.235380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.235381 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.235484 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.235778 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:50.235783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:50.235890 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.235964 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.236088 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":67,"threads_started":1}
I20260810 23:05:50.236250 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.226833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:43801 (local address 127.23.56.126:38651)
0810 23:05:50.226925 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.226928 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.227054 (+ 126us) server_negotiation.cc:407] Connection header received
0810 23:05:50.227576 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.227577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.227588 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.227640 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:50.228541 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.229030 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.230528 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.230719 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.230773 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.233174 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:50.233176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:50.233177 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:50.233186 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:50.234800 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.235228 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.235229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.235230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.235297 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.235563 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.235564 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.235564 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.235711 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:50.235795 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.236032 (+ 237us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.236131 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 23:05:50.236768 7357 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38651
I20260810 23:05:50.236893 7357 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:50.237107 7357 heartbeater.cc:511] Master 127.23.56.126:38651 requested a full tablet report, sending...
I20260810 23:05:50.237468 6978 ts_manager.cc:194] Registered new tserver with Master: 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
I20260810 23:05:50.237905 6978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:43801
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:05:50.259275 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38651
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:50.342476 7363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:50.342667 7363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:50.342689 7363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:50.343837 7363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:50.343883 7363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:50.343910 7363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:05:50.345359 7363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:50.345594 7363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:50.345824 7363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:50.350798 7369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.350771 7371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.350708 7368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:50.350836 7363 server_base.cc:1034] running on GCE node
I20260810 23:05:50.351190 7363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:50.351413 7363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:50.352550 7363 hybrid_clock.cc:648] HybridClock initialized: now 1786403150352538 us; error 27 us; skew 500 ppm
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:50.354984 7363 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:05:50.355489 7363 webserver.cc:460] Webserver started at http://127.23.56.67:34725/ using document root <none> and password file <none>
I20260810 23:05:50.355700 7363 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:50.355743 7363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:50.355887 7363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:05:50.356915 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "70b852721ea34202b33a3942096ebf3e"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "09a98880d29b27805048d3cb420a5cea"
server_key_iv: "c56ed53c8ab7cb46a4ac4226d4c1d8a8"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.357270 7363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "70b852721ea34202b33a3942096ebf3e"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "09a98880d29b27805048d3cb420a5cea"
server_key_iv: "c56ed53c8ab7cb46a4ac4226d4c1d8a8"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.358566 7363 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:05:50.359304 7379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.359460 7363 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:05:50.359551 7363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "70b852721ea34202b33a3942096ebf3e"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "09a98880d29b27805048d3cb420a5cea"
server_key_iv: "c56ed53c8ab7cb46a4ac4226d4c1d8a8"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.359602 7363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:50.382378 7363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:50.384748 7363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:50.384859 7363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:50.385056 7363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:50.385409 7363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:50.385445 7363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.385490 7363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:50.385520 7363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.391597 7363 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:41095
I20260810 23:05:50.391702 7510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:41095 every 8 connection(s)
I20260810 23:05:50.392020 7363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:50.394863 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 7363
I20260810 23:05:50.394939 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:05:50.395229 23777 external_mini_cluster.cc:1428] Setting key 2383a2aaf8b10daa7a62f9e1682076c0
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:50.401476 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.392763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:46759 (local address 127.23.56.126:38651)
0810 23:05:50.392830 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.392832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.393089 (+ 257us) server_negotiation.cc:407] Connection header received
0810 23:05:50.393528 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.393529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.393541 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.393587 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:50.394454 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.395053 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.396412 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.396597 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.396653 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.398716 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:50.398718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:50.398718 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:50.398743 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:50.400226 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.400585 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.400586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.400586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.400636 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.400939 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.400940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.400941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.401045 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:50.401094 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.401342 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.401414 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:05:50.401589 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.392740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:46759)
0810 23:05:50.392942 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.392977 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.393448 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.393644 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.393647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.394181 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:50.394370 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.394375 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.395169 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.395171 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.396336 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.396339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.396671 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.396672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.396765 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.397243 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:50.397256 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:50.398635 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:50.400297 (+ 1662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.400302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.400304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.400521 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.400741 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.400742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.400742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.400886 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.401100 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:50.401105 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:50.401206 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.401286 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.401392 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":117,"threads_started":1}
I20260810 23:05:50.402060 7511 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38651
I20260810 23:05:50.402148 7511 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:50.402348 7511 heartbeater.cc:511] Master 127.23.56.126:38651 requested a full tablet report, sending...
I20260810 23:05:50.402704 6978 ts_manager.cc:194] Registered new tserver with Master: 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095)
I20260810 23:05:50.403077 6978 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:46759
I20260810 23:05:50.406841 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:50.413606 7024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.408195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40014 (local address 127.23.56.126:38651)
0810 23:05:50.408237 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.408239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.408376 (+ 137us) server_negotiation.cc:407] Connection header received
0810 23:05:50.408412 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.408412 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.408421 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.408465 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:50.408992 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.409561 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.410205 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.410385 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.410434 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.411605 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:50.411606 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:50.411607 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:50.411613 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:50.412700 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.412985 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.412985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.412986 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.413031 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.413187 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.413187 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.413187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.413318 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:50.413390 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.413468 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.413521 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:05:50.415267 6978 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:05:50.416807 6978 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:05:50.423039 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.418924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:40009 (local address 127.0.0.1:44524)
0810 23:05:50.419242 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.419282 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.419347 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.419651 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.419652 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.419654 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:50.419698 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.419699 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.421080 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.421082 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.421976 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.421978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.422744 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.422745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.422822 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.422833 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.422874 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.422927 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":40,"thread_start_us":51,"threads_started":1}
I20260810 23:05:50.423066 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.419030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43938 (local address 127.23.56.65:35609)
0810 23:05:50.419388 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.419392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.419404 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:05:50.419448 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.419450 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.419466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.419553 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:50.419728 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.420662 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.422005 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.422634 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.422722 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.422782 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.422935 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.422987 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":112,"threads_started":1}
I20260810 23:05:50.423461 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.419212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.23.56.67:41095)
0810 23:05:50.419397 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.419401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.419771 (+ 370us) server_negotiation.cc:407] Connection header received
0810 23:05:50.419811 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.419813 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.419829 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.419896 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:50.420084 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.421098 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.422383 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.423063 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.423143 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.423207 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.423357 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.423394 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":90,"threads_started":1}
I20260810 23:05:50.423060 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.418916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:35609 (local address 127.0.0.1:43938)
0810 23:05:50.419240 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.419280 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.419346 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.419564 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.419567 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.419568 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:50.419635 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.419636 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.420769 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.420770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.421896 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.421899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.422725 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.422726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.422815 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.422828 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.422919 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.422962 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":49,"threads_started":1}
I20260810 23:05:50.423384 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.419114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41095 (local address 127.0.0.1:37944)
0810 23:05:50.419649 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.419660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.419713 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.419962 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.419963 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.419965 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:50.420004 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.420006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.421197 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.421198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.422299 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.422301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.423222 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.423223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.423271 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.423274 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.423318 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.423344 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":67,"threads_started":1}
I20260810 23:05:50.423913 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.419171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.23.56.66:40009)
0810 23:05:50.419491 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.419495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.419504 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:05:50.419544 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.419546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.419561 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.419640 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:50.419759 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.420966 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.422069 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.422648 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.423679 (+ 1031us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.423747 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.423768 (+ 21us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.423818 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":71,"threads_started":1}
I20260810 23:05:50.424315 7126 tablet_service.cc:1467] Processing CreateTablet for tablet a52f83a3dbd74472982219c964fd40f0 (DEFAULT_TABLE table=test-table [id=fe545d91fd1b4fab82572d9a45d7d617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:50.424607 7126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52f83a3dbd74472982219c964fd40f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:50.426967 7526 tablet_bootstrap.cc:492] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Bootstrap starting.
I20260810 23:05:50.427338 7439 tablet_service.cc:1467] Processing CreateTablet for tablet a52f83a3dbd74472982219c964fd40f0 (DEFAULT_TABLE table=test-table [id=fe545d91fd1b4fab82572d9a45d7d617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:50.427618 7439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52f83a3dbd74472982219c964fd40f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:50.427680 7526 tablet_bootstrap.cc:654] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:50.428154 7526 log.cc:824] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:50.428942 7526 tablet_bootstrap.cc:492] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: No bootstrap required, opened a new log
I20260810 23:05:50.428841 7278 tablet_service.cc:1467] Processing CreateTablet for tablet a52f83a3dbd74472982219c964fd40f0 (DEFAULT_TABLE table=test-table [id=fe545d91fd1b4fab82572d9a45d7d617]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:50.429015 7526 ts_tablet_manager.cc:1397] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:05:50.429173 7278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a52f83a3dbd74472982219c964fd40f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:50.430245 7528 tablet_bootstrap.cc:492] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: Bootstrap starting.
I20260810 23:05:50.430860 7528 tablet_bootstrap.cc:654] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:50.430945 7526 raft_consensus.cc:348] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.431113 7526 raft_consensus.cc:374] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:50.431186 7526 raft_consensus.cc:729] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d079a63756c84a569fe91f0fbeaeadd3, State: Initialized, Role: FOLLOWER
I20260810 23:05:50.431334 7526 consensus_queue.cc:260] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.431511 7528 log.cc:824] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:50.431635 7526 ts_tablet_manager.cc:1428] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260810 23:05:50.431771 7201 heartbeater.cc:503] Master 127.23.56.126:38651 was elected leader, sending a full tablet report...
I20260810 23:05:50.432265 7528 tablet_bootstrap.cc:492] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: No bootstrap required, opened a new log
I20260810 23:05:50.432348 7528 ts_tablet_manager.cc:1397] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 23:05:50.432530 7531 tablet_bootstrap.cc:492] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: Bootstrap starting.
I20260810 23:05:50.433145 7531 tablet_bootstrap.cc:654] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:50.433676 7531 log.cc:824] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:50.434432 7531 tablet_bootstrap.cc:492] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: No bootstrap required, opened a new log
I20260810 23:05:50.434506 7531 ts_tablet_manager.cc:1397] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:05:50.434441 7528 raft_consensus.cc:348] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.434605 7528 raft_consensus.cc:374] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:50.434648 7528 raft_consensus.cc:729] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b852721ea34202b33a3942096ebf3e, State: Initialized, Role: FOLLOWER
I20260810 23:05:50.434749 7528 consensus_queue.cc:260] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.434996 7528 ts_tablet_manager.cc:1428] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 23:05:50.435022 7511 heartbeater.cc:503] Master 127.23.56.126:38651 was elected leader, sending a full tablet report...
I20260810 23:05:50.436339 7531 raft_consensus.cc:348] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.436506 7531 raft_consensus.cc:374] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:50.436542 7531 raft_consensus.cc:729] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a6dc92f3634db384160aa638dc3f9d, State: Initialized, Role: FOLLOWER
I20260810 23:05:50.436667 7531 consensus_queue.cc:260] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.436921 7531 ts_tablet_manager.cc:1428] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:05:50.437007 7357 heartbeater.cc:503] Master 127.23.56.126:38651 was elected leader, sending a full tablet report...
W20260810 23:05:50.476655 7359 tablet.cc:2369] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:05:50.506273 7533 raft_consensus.cc:482] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:50.506398 7533 raft_consensus.cc:504] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.506744 7533 leader_election.cc:288] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d079a63756c84a569fe91f0fbeaeadd3 (127.23.56.65:35609), 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
I20260810 23:05:50.509635 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.507000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:39901 (local address 127.23.56.65:35609)
0810 23:05:50.507061 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.507064 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.507087 (+ 23us) server_negotiation.cc:407] Connection header received
0810 23:05:50.507136 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.507137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.507150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.507208 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:50.507387 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.507902 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.508904 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.509277 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.509360 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.509370 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.509534 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.509571 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 23:05:50.509634 7514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.506901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:35609 (local address 127.23.56.67:39901)
0810 23:05:50.506970 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.506981 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.507026 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.507236 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.507237 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.507239 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:50.507317 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.507318 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.507980 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.507981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.508812 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.508814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.509359 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.509360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.509442 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.509445 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.509522 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.509562 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":35}
I20260810 23:05:50.509712 7524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.507031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:44857 (local address 127.23.56.66:40009)
0810 23:05:50.507079 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.507081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.507274 (+ 193us) server_negotiation.cc:407] Connection header received
0810 23:05:50.507331 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.507332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.507343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.507409 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:50.507578 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.508059 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.508988 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.509346 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.509424 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.509432 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.509587 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.509623 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 23:05:50.509758 7535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.506931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:40009 (local address 127.23.56.67:44857)
0810 23:05:50.507215 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.507225 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.507281 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.507415 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.507416 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.507418 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:50.507492 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.507494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.508143 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.508144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.508897 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.508900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.509435 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.509436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.509514 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.509517 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.509580 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.509619 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":120,"threads_started":1}
I20260810 23:05:50.510056 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52f83a3dbd74472982219c964fd40f0" candidate_uuid: "70b852721ea34202b33a3942096ebf3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d079a63756c84a569fe91f0fbeaeadd3" is_pre_election: true
I20260810 23:05:50.510185 7149 raft_consensus.cc:2454] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b852721ea34202b33a3942096ebf3e in term 0.
I20260810 23:05:50.510393 7383 leader_election.cc:302] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70b852721ea34202b33a3942096ebf3e, d079a63756c84a569fe91f0fbeaeadd3; no voters:
I20260810 23:05:50.510389 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52f83a3dbd74472982219c964fd40f0" candidate_uuid: "70b852721ea34202b33a3942096ebf3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32a6dc92f3634db384160aa638dc3f9d" is_pre_election: true
I20260810 23:05:50.510525 7305 raft_consensus.cc:2454] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70b852721ea34202b33a3942096ebf3e in term 0.
I20260810 23:05:50.510546 7533 raft_consensus.cc:2781] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:50.510622 7533 raft_consensus.cc:482] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:50.510654 7533 raft_consensus.cc:3037] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:50.511401 7533 raft_consensus.cc:504] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.511554 7533 leader_election.cc:288] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [CANDIDATE]: Term 1 election: Requested vote from peers d079a63756c84a569fe91f0fbeaeadd3 (127.23.56.65:35609), 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
I20260810 23:05:50.511729 7305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52f83a3dbd74472982219c964fd40f0" candidate_uuid: "70b852721ea34202b33a3942096ebf3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32a6dc92f3634db384160aa638dc3f9d"
I20260810 23:05:50.511770 7149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a52f83a3dbd74472982219c964fd40f0" candidate_uuid: "70b852721ea34202b33a3942096ebf3e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d079a63756c84a569fe91f0fbeaeadd3"
I20260810 23:05:50.511849 7149 raft_consensus.cc:3037] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:50.511855 7305 raft_consensus.cc:3037] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:50.512557 7149 raft_consensus.cc:2454] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70b852721ea34202b33a3942096ebf3e in term 1.
I20260810 23:05:50.512681 7305 raft_consensus.cc:2454] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70b852721ea34202b33a3942096ebf3e in term 1.
I20260810 23:05:50.512712 7383 leader_election.cc:302] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70b852721ea34202b33a3942096ebf3e, d079a63756c84a569fe91f0fbeaeadd3; no voters:
I20260810 23:05:50.512789 7533 raft_consensus.cc:2781] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:50.512955 7533 raft_consensus.cc:686] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 1 LEADER]: Becoming Leader. State: Replica: 70b852721ea34202b33a3942096ebf3e, State: Running, Role: LEADER
I20260810 23:05:50.513067 7533 consensus_queue.cc:237] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.513913 6978 catalog_manager.cc:5495] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e reported cstate change: term changed from 0 to 1, leader changed from <none> to 70b852721ea34202b33a3942096ebf3e (127.23.56.67). New cstate: current_term: 1 leader_uuid: "70b852721ea34202b33a3942096ebf3e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } health_report { overall_health: HEALTHY } } }
I20260810 23:05:50.519834 7525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.517697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37946 (local address 127.23.56.67:41095)
0810 23:05:50.517752 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.517753 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.517761 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:05:50.517852 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.517853 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.517867 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.517923 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:50.518013 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.518421 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.519161 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.519341 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.519390 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.519529 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:50.519612 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:50.519669 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.519721 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.519737 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 23:05:50.521377 7149 raft_consensus.cc:1261] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Refusing update from remote peer 70b852721ea34202b33a3942096ebf3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:50.521400 7305 raft_consensus.cc:1261] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [term 1 FOLLOWER]: Refusing update from remote peer 70b852721ea34202b33a3942096ebf3e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:50.521759 7536 consensus_queue.cc:1035] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:50.521894 7533 consensus_queue.cc:1035] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [LEADER]: Connected to new peer: Peer: permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:50.524505 7540 mvcc.cc:205] Tried to move back new op lower bound from 7317107304533413888 to 7317107304502198272. Current Snapshot: MvccSnapshot[applied={T|T < 7317107304533413888}]
I20260810 23:05:50.524748 7542 mvcc.cc:205] Tried to move back new op lower bound from 7317107304533413888 to 7317107304502198272. Current Snapshot: MvccSnapshot[applied={T|T < 7317107304533413888}]
I20260810 23:05:50.526027 6978 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:50.526113 6978 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:50.526880 6978 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Sending DeleteTablet for 3 replicas of tablet a52f83a3dbd74472982219c964fd40f0
I20260810 23:05:50.527220 7126 tablet_service.cc:1514] Processing DeleteTablet for tablet a52f83a3dbd74472982219c964fd40f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:50 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:43938
I20260810 23:05:50.527247 7439 tablet_service.cc:1514] Processing DeleteTablet for tablet a52f83a3dbd74472982219c964fd40f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:50 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:37944
I20260810 23:05:50.527305 7278 tablet_service.cc:1514] Processing DeleteTablet for tablet a52f83a3dbd74472982219c964fd40f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:50 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:44524
I20260810 23:05:50.527438 7549 tablet_replica.cc:330] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: stopping tablet replica
I20260810 23:05:50.527493 7551 tablet_replica.cc:330] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: stopping tablet replica
I20260810 23:05:50.527534 7549 raft_consensus.cc:2229] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:50.527583 7551 raft_consensus.cc:2229] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:50.527637 7549 raft_consensus.cc:2258] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:50.527683 7551 raft_consensus.cc:2258] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:50.527732 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 7047
I20260810 23:05:50.530467 7551 ts_tablet_manager.cc:1905] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:50.532923 7551 ts_tablet_manager.cc:1918] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:50.533016 7551 log.cc:1196] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/a52f83a3dbd74472982219c964fd40f0
I20260810 23:05:50.533416 7551 ts_tablet_manager.cc:1939] T a52f83a3dbd74472982219c964fd40f0 P 32a6dc92f3634db384160aa638dc3f9d: Deleting consensus metadata
I20260810 23:05:50.533942 6959 catalog_manager.cc:4841] TS 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009): tablet a52f83a3dbd74472982219c964fd40f0 (table test-table [id=fe545d91fd1b4fab82572d9a45d7d617]) successfully deleted
I20260810 23:05:50.536999 7550 tablet_replica.cc:330] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: stopping tablet replica
I20260810 23:05:50.537106 7550 raft_consensus.cc:2229] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:05:50.537225 7550 raft_consensus.cc:2258] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:50.537647 7550 ts_tablet_manager.cc:1905] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:50.539419 7550 ts_tablet_manager.cc:1918] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:50.539502 7550 log.cc:1196] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/a52f83a3dbd74472982219c964fd40f0
I20260810 23:05:50.539784 7550 ts_tablet_manager.cc:1939] T a52f83a3dbd74472982219c964fd40f0 P 70b852721ea34202b33a3942096ebf3e: Deleting consensus metadata
I20260810 23:05:50.540257 6962 catalog_manager.cc:4841] TS 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095): tablet a52f83a3dbd74472982219c964fd40f0 (table test-table [id=fe545d91fd1b4fab82572d9a45d7d617]) successfully deleted
W20260810 23:05:50.541342 6962 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:35609 (error 108)
I20260810 23:05:50.541555 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 7206
I20260810 23:05:50.542932 7523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.541672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:35609 (local address 127.0.0.1:43948)
0810 23:05:50.542862 (+ 1190us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.542901 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1168}
W20260810 23:05:50.544317 6962 catalog_manager.cc:4584] TS d079a63756c84a569fe91f0fbeaeadd3 (127.23.56.65:35609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a52f83a3dbd74472982219c964fd40f0: Network error: Client connection negotiation failed: client connection to 127.23.56.65:35609: connect: Connection refused (error 111)
I20260810 23:05:50.557053 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 7363
I20260810 23:05:50.563436 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 6939
I20260810 23:05:50.570928 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38651
--webserver_interface=127.23.56.126
--webserver_port=40961
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:38651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:50.651381 7553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:50.651616 7553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:50.651651 7553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:50.652740 7553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:50.652808 7553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:50.652835 7553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:50.652868 7553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:50.652896 7553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:50.654384 7553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:38651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=40961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:50.654666 7553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:50.654884 7553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:50.665930 7562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.665932 7560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.665930 7559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:50.666285 7553 server_base.cc:1034] running on GCE node
I20260810 23:05:50.666534 7553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:50.666852 7553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:50.668017 7553 hybrid_clock.cc:648] HybridClock initialized: now 1786403150668010 us; error 37 us; skew 500 ppm
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:50.670629 7553 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:05:50.671074 7553 webserver.cc:460] Webserver started at http://127.23.56.126:40961/ using document root <none> and password file <none>
I20260810 23:05:50.671360 7553 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:50.671445 7553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:50.672868 7553 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:05:50.673523 7570 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.673704 7553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:05:50.673846 7553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "314daad9919d4a7887cb9397cfa6e6f3"
format_stamp: "Formatted at 2026-08-10 23:05:49 on dist-test-slave-1xrh"
server_key: "08531869c8264b2fc97dbae4a875c376"
server_key_iv: "76b5998b7eb99e02aa944601a1a2301d"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.674245 7553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:50.682039 7553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:50.683873 7553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:50.684022 7553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:50.687939 7553 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:38651
I20260810 23:05:50.687969 7628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:38651 every 8 connection(s)
I20260810 23:05:50.688365 7553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:50.689952 7629 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:05:50.690604 7629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Bootstrap starting.
I20260810 23:05:50.695933 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 7553
I20260810 23:05:50.696214 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:35609
--local_ip_for_outbound_sockets=127.23.56.65
--tserver_master_addrs=127.23.56.126:38651
--webserver_port=33653
--webserver_interface=127.23.56.65
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:50.699251 7629 log.cc:824] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:50.701220 7629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:05:50.701531 7629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Bootstrap complete.
I20260810 23:05:50.703970 7629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:50.704233 7629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 314daad9919d4a7887cb9397cfa6e6f3, State: Initialized, Role: FOLLOWER
I20260810 23:05:50.704442 7629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:50.704541 7629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:50.704602 7629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:50.704650 7629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:05:50.705658 7629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:50.705799 7629 leader_election.cc:302] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 314daad9919d4a7887cb9397cfa6e6f3; no voters:
I20260810 23:05:50.705999 7629 leader_election.cc:288] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:05:50.706035 7633 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:05:50.706146 7633 raft_consensus.cc:686] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 2 LEADER]: Becoming Leader. State: Replica: 314daad9919d4a7887cb9397cfa6e6f3, State: Running, Role: LEADER
I20260810 23:05:50.706274 7629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:50.706303 7633 consensus_queue.cc:237] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:50.706668 7635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "314daad9919d4a7887cb9397cfa6e6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } } }
I20260810 23:05:50.706681 7636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 314daad9919d4a7887cb9397cfa6e6f3. Latest consensus state: current_term: 2 leader_uuid: "314daad9919d4a7887cb9397cfa6e6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } } }
I20260810 23:05:50.706776 7635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:50.706786 7636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:50.707003 7641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:50.707939 7641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:50.708048 7641 catalog_manager.cc:1259] Loaded cluster ID: e44427ddb0f34cb3b45fdd4ce63a66b7
I20260810 23:05:50.708083 7641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:05:50.708317 7641 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 23:05:50.713054 7641 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 23:05:50.715435 7641 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:05:50.715903 7641 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Loaded TSK: 0
I20260810 23:05:50.716123 7641 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:05:50.781229 7631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:50.781522 7631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:50.781566 7631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:50.782737 7631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:50.782812 7631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:50.782862 7631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:05:50.784400 7631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:35609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=33653
--tserver_master_addrs=127.23.56.126:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:50.784682 7631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:50.784932 7631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:50.790167 7661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.790179 7657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.790186 7658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:50.790817 7631 server_base.cc:1034] running on GCE node
I20260810 23:05:50.791044 7631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:50.791363 7631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:50.792551 7631 hybrid_clock.cc:648] HybridClock initialized: now 1786403150792531 us; error 34 us; skew 500 ppm
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:50.795761 7631 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:05:50.796278 7631 webserver.cc:460] Webserver started at http://127.23.56.65:33653/ using document root <none> and password file <none>
I20260810 23:05:50.796566 7631 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:50.796665 7631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:50.798213 7631 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260810 23:05:50.799032 7670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.799245 7631 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:50.799371 7631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "d079a63756c84a569fe91f0fbeaeadd3"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "c46e2ca90106b71c332cebc92e9a693a"
server_key_iv: "754947cfa6730b5cd125c553633bef4f"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.799747 7631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:50.812040 7631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:50.814536 7631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:50.814709 7631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:50.814952 7631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:50.815528 7677 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:05:50.817041 7631 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:05:50.817116 7631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 23:05:50.817143 7631 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:05:50.817854 7631 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:05:50.817904 7631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 23:05:50.817956 7677 tablet_bootstrap.cc:492] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Bootstrap starting.
I20260810 23:05:50.826330 7631 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:35609
I20260810 23:05:50.826463 7800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:35609 every 8 connection(s)
I20260810 23:05:50.826866 7631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:50.830336 7677 log.cc:824] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:50.832250 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 7631
I20260810 23:05:50.832532 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:40009
--local_ip_for_outbound_sockets=127.23.56.66
--tserver_master_addrs=127.23.56.126:38651
--webserver_port=41609
--webserver_interface=127.23.56.66
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:50.835592 7677 tablet_bootstrap.cc:492] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:05:50.835886 7677 tablet_bootstrap.cc:492] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Bootstrap complete.
I20260810 23:05:50.837231 7677 ts_tablet_manager.cc:1397] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.005s
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:50.838474 7677 raft_consensus.cc:348] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.838799 7677 raft_consensus.cc:729] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d079a63756c84a569fe91f0fbeaeadd3, State: Initialized, Role: FOLLOWER
I20260810 23:05:50.838943 7677 consensus_queue.cc:260] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } }
I20260810 23:05:50.839241 7677 ts_tablet_manager.cc:1428] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260810 23:05:50.843879 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.831258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:54403 (local address 127.23.56.126:38651)
0810 23:05:50.831493 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.831501 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.831542 (+ 41us) server_negotiation.cc:407] Connection header received
0810 23:05:50.832236 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.832241 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.832288 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.832445 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:50.833296 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.834421 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.836012 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.836469 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.836633 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.839668 (+ 3035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:50.839672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:50.839673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:50.839684 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:50.841710 (+ 2026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.842333 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.842337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.842337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.842449 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.842779 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.842780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.842780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.843074 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:50.843188 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.843623 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.843730 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":108,"threads_started":1}
I20260810 23:05:50.844012 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.831077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:54403)
0810 23:05:50.831388 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.831434 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.831989 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.832601 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.832606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.832970 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:50.833164 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.833169 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.834574 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.834576 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.835910 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.835913 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.836598 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.836600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.836754 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.837415 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:50.837426 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:50.839543 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:50.841819 (+ 2276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.841827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.841830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.842199 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.842530 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.842531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.842532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.842685 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.843196 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:50.843203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:50.843383 (+ 180us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.843516 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.843717 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":129,"threads_started":1}
I20260810 23:05:50.844790 7801 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38651
I20260810 23:05:50.844913 7801 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:50.845141 7801 heartbeater.cc:511] Master 127.23.56.126:38651 requested a full tablet report, sending...
I20260810 23:05:50.845904 7590 ts_manager.cc:194] Registered new tserver with Master: d079a63756c84a569fe91f0fbeaeadd3 (127.23.56.65:35609)
I20260810 23:05:50.846812 7590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:54403
I20260810 23:05:50.847961 7801 heartbeater.cc:503] Master 127.23.56.126:38651 was elected leader, sending a full tablet report...
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:05:50.854712 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.846771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43950 (local address 127.23.56.65:35609)
0810 23:05:50.846964 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.846968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.846987 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:05:50.847018 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.847019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.847033 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.847101 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:50.847527 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.848267 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.849491 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.849752 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.849803 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.851919 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:50.851927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:50.851927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:50.851934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:50.853405 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.853864 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.853867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.853867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.853932 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.854137 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.854138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.854138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.854267 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:50.854356 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.854532 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.854617 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260810 23:05:50.854712 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.846494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:35609 (local address 127.0.0.1:43950)
0810 23:05:50.846689 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.846723 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.846787 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.847103 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.847107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.847398 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:50.847436 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.847438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.848361 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.848363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.849371 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.849373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.849834 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.849835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.849917 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.850349 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:50.850355 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:50.851832 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:50.853534 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.853541 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.853543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.853752 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.854001 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.854002 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.854002 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.854067 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.854338 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:50.854345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:50.854394 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.854493 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.854609 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":98,"threads_started":1}
I20260810 23:05:50.855315 7724 tablet_service.cc:1514] Processing DeleteTablet for tablet a52f83a3dbd74472982219c964fd40f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:50 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:43950
I20260810 23:05:50.855787 7810 tablet_replica.cc:330] stopping tablet replica
I20260810 23:05:50.855895 7810 raft_consensus.cc:2229] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:50.855957 7810 raft_consensus.cc:2258] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:50.856240 7810 ts_tablet_manager.cc:1905] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:50.858696 7810 ts_tablet_manager.cc:1918] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:50.858773 7810 log.cc:1196] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/a52f83a3dbd74472982219c964fd40f0
I20260810 23:05:50.859036 7810 ts_tablet_manager.cc:1939] T a52f83a3dbd74472982219c964fd40f0 P d079a63756c84a569fe91f0fbeaeadd3: Deleting consensus metadata
I20260810 23:05:50.859529 7573 catalog_manager.cc:4841] TS d079a63756c84a569fe91f0fbeaeadd3 (127.23.56.65:35609): tablet a52f83a3dbd74472982219c964fd40f0 (table test-table [id=fe545d91fd1b4fab82572d9a45d7d617]) successfully deleted
W20260810 23:05:50.929926 7806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:50.930120 7806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:50.930145 7806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:50.931383 7806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:50.931425 7806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:50.931455 7806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:05:50.932955 7806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:40009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=41609
--tserver_master_addrs=127.23.56.126:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:50.933133 7806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:50.933324 7806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:50.938571 7817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.938589 7819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:50.938584 7816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:50.938920 7806 server_base.cc:1034] running on GCE node
I20260810 23:05:50.939117 7806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:50.939435 7806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:50.940590 7806 hybrid_clock.cc:648] HybridClock initialized: now 1786403150940568 us; error 36 us; skew 500 ppm
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:50.943171 7806 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:05:50.943678 7806 webserver.cc:460] Webserver started at http://127.23.56.66:41609/ using document root <none> and password file <none>
I20260810 23:05:50.943954 7806 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:50.944041 7806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:50.945415 7806 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:05:50.946009 7827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.946183 7806 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:05:50.946285 7806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "32a6dc92f3634db384160aa638dc3f9d"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "a7173a3e7dea2bc38b8372b7bc5e6033"
server_key_iv: "db2269be9998f160b0238ab056ea54d6"
server_key_version: "encryptionkey@0"
I20260810 23:05:50.946628 7806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:50.966902 7806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:50.968848 7806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:50.969061 7806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:50.969326 7806 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:50.969660 7806 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:50.969717 7806 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.969754 7806 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:50.969785 7806 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:50.976015 7806 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:40009
I20260810 23:05:50.976051 7960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:40009 every 8 connection(s)
I20260810 23:05:50.976428 7806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:50.981010 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 7806
I20260810 23:05:50.981316 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:41095
--local_ip_for_outbound_sockets=127.23.56.67
--tserver_master_addrs=127.23.56.126:38651
--webserver_port=34725
--webserver_interface=127.23.56.67
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 23:05:50 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403150, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:50.986330 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.977172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:34183 (local address 127.23.56.126:38651)
0810 23:05:50.977219 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:50.977221 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:50.977462 (+ 241us) server_negotiation.cc:407] Connection header received
0810 23:05:50.978108 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:50.978110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:50.978123 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:50.978171 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:50.979094 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.979608 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.980632 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.980882 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.980968 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.983161 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:50.983162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:50.983162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:50.983170 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:50.984835 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.985249 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.985250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.985251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.985313 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:50.985553 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:50.985555 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:50.985555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:50.985711 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:50.985785 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:50.986098 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 23:05:50.986197 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:05:50.986642 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:50.977200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:34183)
0810 23:05:50.977342 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:50.977376 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:50.977964 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:50.978253 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:50.978257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:50.978813 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:50.978995 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.978999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.979710 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.979712 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.980528 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:50.980530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:50.980980 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:50.980981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:50.981119 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:50.981671 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:50.981687 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:50.983048 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:50.984922 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.984928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.984930 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.985172 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.985382 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:50.985383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:50.985383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:50.985487 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:50.985781 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:50.985785 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:50.985935 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 23:05:50.986064 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 23:05:50.986193 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260810 23:05:50.987200 7961 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38651
I20260810 23:05:50.987303 7961 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:50.987509 7961 heartbeater.cc:511] Master 127.23.56.126:38651 requested a full tablet report, sending...
I20260810 23:05:50.987946 7590 ts_manager.cc:194] Registered new tserver with Master: 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
I20260810 23:05:50.988471 7590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:34183
W20260810 23:05:51.062835 7964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:51.063012 7964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:51.063035 7964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:51.064168 7964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:51.064213 7964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:51.064240 7964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:05:51.065595 7964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:41095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=34725
--tserver_master_addrs=127.23.56.126:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:51.065757 7964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:51.065927 7964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:51.070945 7973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:51.071013 7970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:51.071092 7964 server_base.cc:1034] running on GCE node
W20260810 23:05:51.070956 7971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:51.071447 7964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:51.071684 7964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:51.072868 7964 hybrid_clock.cc:648] HybridClock initialized: now 1786403151072861 us; error 35 us; skew 500 ppm
Aug 10 23:05:51 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403151, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:51.075289 7964 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:05:51.075712 7964 webserver.cc:460] Webserver started at http://127.23.56.67:34725/ using document root <none> and password file <none>
I20260810 23:05:51.075953 7964 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:51.076035 7964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:51.077409 7964 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:51.078002 7982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:51.078181 7964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:05:51.078276 7964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "70b852721ea34202b33a3942096ebf3e"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "09a98880d29b27805048d3cb420a5cea"
server_key_iv: "c56ed53c8ab7cb46a4ac4226d4c1d8a8"
server_key_version: "encryptionkey@0"
I20260810 23:05:51.078598 7964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:51.099650 7964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:51.101460 7964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:51.101621 7964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:51.101861 7964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:51.102227 7964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:51.102281 7964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:51.102316 7964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:51.102337 7964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:51.108559 7964 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:41095
I20260810 23:05:51.108639 8108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:41095 every 8 connection(s)
I20260810 23:05:51.108997 7964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:05:51 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403151, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:05:51.117822 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 7964
I20260810 23:05:51.117892 8111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.109679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:60549)
0810 23:05:51.109835 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.109871 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:51.110386 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:51.110591 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:51.110594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:51.111227 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:51.111412 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.111417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.112068 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.112068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.112836 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.112838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.113166 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.113167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.113263 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.113815 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:51.113824 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:51.115154 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:51.116668 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:51.116674 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:51.116675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:51.116902 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:51.117095 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:51.117096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:51.117096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:51.117184 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:51.117419 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:51.117423 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:51.117524 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 23:05:51.117615 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:05:51.117727 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260810 23:05:51.117933 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.109725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:60549 (local address 127.23.56.126:38651)
0810 23:05:51.109803 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.109806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.109975 (+ 169us) server_negotiation.cc:407] Connection header received
0810 23:05:51.110473 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.110474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.110484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.110532 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:51.111497 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.111988 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.112913 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.113095 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.113144 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.115223 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:51.115225 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:51.115225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:51.115233 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:51.116592 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:51.116973 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:51.116974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:51.116974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:51.117033 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:51.117240 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:51.117241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:51.117241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:51.117363 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:51.117418 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.117709 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.117804 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:05:51.118472 8109 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38651
I20260810 23:05:51.118571 8109 heartbeater.cc:464] Registering TS with master...
I20260810 23:05:51.118752 8109 heartbeater.cc:511] Master 127.23.56.126:38651 requested a full tablet report, sending...
I20260810 23:05:51.119275 7590 ts_manager.cc:194] Registered new tserver with Master: 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095)
I20260810 23:05:51.119668 7590 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:60549
I20260810 23:05:51.123870 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.118233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40018 (local address 127.23.56.126:38651)
0810 23:05:51.118335 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.118337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.118397 (+ 60us) server_negotiation.cc:407] Connection header received
0810 23:05:51.118500 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.118501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.118511 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.118589 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:51.119319 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.119876 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.120583 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.120752 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.120806 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.121829 (+ 1023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:51.121830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:51.121830 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:51.121836 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:51.122965 (+ 1129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:51.123332 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:51.123333 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:51.123333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:51.123376 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:51.123494 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:51.123495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:51.123495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:51.123583 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:51.123689 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.123734 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.123796 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":71,"threads_started":1}
I20260810 23:05:51.124518 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:05:51.126925 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.124868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40034 (local address 127.23.56.126:38651)
0810 23:05:51.124913 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.124914 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.125007 (+ 93us) server_negotiation.cc:407] Connection header received
0810 23:05:51.125058 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.125059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.125081 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.125133 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:51.125197 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.125594 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.126310 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.126471 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.126525 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.126652 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:51.126732 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:51.126785 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.126831 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.126843 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:05:51.127228 7590 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:40034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:05:51.128858 7590 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:05:51.131085 7724 tablet_service.cc:1467] Processing CreateTablet for tablet 8e714ef4a2bc4ab8841b458d2e987ac0 (DEFAULT_TABLE table=test-table [id=441eb506436c405da910faebce30f10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:51.131325 7724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e714ef4a2bc4ab8841b458d2e987ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:51.132855 7677 tablet_bootstrap.cc:492] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Bootstrap starting.
I20260810 23:05:51.133242 7677 tablet_bootstrap.cc:654] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:51.133894 7677 tablet_bootstrap.cc:492] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: No bootstrap required, opened a new log
I20260810 23:05:51.133947 7677 ts_tablet_manager.cc:1397] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:05:51.134346 7677 raft_consensus.cc:348] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.134445 7677 raft_consensus.cc:374] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:51.134501 7677 raft_consensus.cc:729] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d079a63756c84a569fe91f0fbeaeadd3, State: Initialized, Role: FOLLOWER
I20260810 23:05:51.134559 7677 consensus_queue.cc:260] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.134677 7677 ts_tablet_manager.cc:1428] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 23:05:51.134935 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.130712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41095 (local address 127.0.0.1:37958)
0810 23:05:51.130897 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.130909 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:51.130947 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:51.131243 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:51.131244 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:51.131246 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:51.131288 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.131289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.132527 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.132528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.133811 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.133815 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.134651 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.134653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.134757 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.134778 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 23:05:51.134812 (+ 34us) client_negotiation.cc:240] Negotiation successful
0810 23:05:51.134855 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":36}
I20260810 23:05:51.135053 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.130727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:40009 (local address 127.0.0.1:44532)
0810 23:05:51.131039 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.131048 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:51.131095 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:51.131494 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:51.131495 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:51.131497 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:51.131544 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.131545 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.132758 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.132760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.133668 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.133671 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.134836 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.134836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.134888 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.134891 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:05:51.134931 (+ 40us) client_negotiation.cc:240] Negotiation successful
0810 23:05:51.134970 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":83,"threads_started":1}
I20260810 23:05:51.135273 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.130866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.23.56.67:41095)
0810 23:05:51.131036 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.131040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.131050 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:05:51.131083 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.131085 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.131099 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.131365 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:51.131425 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.132415 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.133904 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.134561 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.135053 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.135117 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.135164 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.135206 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":73,"threads_started":1}
I20260810 23:05:51.135509 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.130841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44532 (local address 127.23.56.66:40009)
0810 23:05:51.131251 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.131264 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.131274 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:05:51.131316 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.131318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.131334 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.131494 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:51.131612 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.132641 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.133767 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.134505 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.134579 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.135184 (+ 605us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.135208 (+ 24us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.135248 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":109,"threads_started":1}
I20260810 23:05:51.135910 8034 tablet_service.cc:1467] Processing CreateTablet for tablet 8e714ef4a2bc4ab8841b458d2e987ac0 (DEFAULT_TABLE table=test-table [id=441eb506436c405da910faebce30f10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:51.136058 7887 tablet_service.cc:1467] Processing CreateTablet for tablet 8e714ef4a2bc4ab8841b458d2e987ac0 (DEFAULT_TABLE table=test-table [id=441eb506436c405da910faebce30f10f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:05:51.136276 8034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e714ef4a2bc4ab8841b458d2e987ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:51.136392 7887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e714ef4a2bc4ab8841b458d2e987ac0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:05:51.138947 8120 tablet_bootstrap.cc:492] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: Bootstrap starting.
I20260810 23:05:51.139174 8122 tablet_bootstrap.cc:492] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: Bootstrap starting.
I20260810 23:05:51.139746 8122 tablet_bootstrap.cc:654] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:51.139768 8120 tablet_bootstrap.cc:654] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: Neither blocks nor log segments found. Creating new log.
I20260810 23:05:51.140192 8122 log.cc:824] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:51.140270 8120 log.cc:824] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:51.141011 8120 tablet_bootstrap.cc:492] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: No bootstrap required, opened a new log
I20260810 23:05:51.141018 8122 tablet_bootstrap.cc:492] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: No bootstrap required, opened a new log
I20260810 23:05:51.141065 8120 ts_tablet_manager.cc:1397] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:05:51.141099 8122 ts_tablet_manager.cc:1397] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 23:05:51.142333 8120 raft_consensus.cc:348] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.142447 8120 raft_consensus.cc:374] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:51.142467 8120 raft_consensus.cc:729] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32a6dc92f3634db384160aa638dc3f9d, State: Initialized, Role: FOLLOWER
I20260810 23:05:51.142588 8120 consensus_queue.cc:260] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.142547 8122 raft_consensus.cc:348] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.142743 8122 raft_consensus.cc:374] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:05:51.142793 8122 raft_consensus.cc:729] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70b852721ea34202b33a3942096ebf3e, State: Initialized, Role: FOLLOWER
I20260810 23:05:51.142894 8120 ts_tablet_manager.cc:1428] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:05:51.142915 8122 consensus_queue.cc:260] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.143013 7961 heartbeater.cc:503] Master 127.23.56.126:38651 was elected leader, sending a full tablet report...
I20260810 23:05:51.143249 8109 heartbeater.cc:503] Master 127.23.56.126:38651 was elected leader, sending a full tablet report...
I20260810 23:05:51.143261 8122 ts_tablet_manager.cc:1428] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260810 23:05:51.227026 7962 tablet.cc:2369] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:05:51.331310 7802 tablet.cc:2369] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:05:51.359596 8110 tablet.cc:2369] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:05:51.417629 8125 raft_consensus.cc:482] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:51.417798 8125 raft_consensus.cc:504] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.418192 8125 leader_election.cc:288] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095), d079a63756c84a569fe91f0fbeaeadd3 (127.23.56.65:35609)
I20260810 23:05:51.422237 8129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.418623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:52559 (local address 127.23.56.65:35609)
0810 23:05:51.418743 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.418745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.418820 (+ 75us) server_negotiation.cc:407] Connection header received
0810 23:05:51.418875 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.418877 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.418925 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.419007 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:51.419154 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.419870 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.421260 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.421900 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.421968 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.422044 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.422120 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.422164 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":64,"threads_started":1}
I20260810 23:05:51.422235 8117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.418526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:37833 (local address 127.23.56.67:41095)
0810 23:05:51.418645 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.418647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.418659 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:05:51.418690 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.418691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.418700 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.418790 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:05:51.418955 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.419763 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.421260 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.421853 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.421940 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.421951 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.422122 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.422164 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 23:05:51.422271 7963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.418414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41095 (local address 127.23.56.66:37833)
0810 23:05:51.418503 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.418516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:51.418563 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:51.418789 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:51.418791 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:51.418793 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:51.418873 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.418875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.419886 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.419887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.421146 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.421150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.421958 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.421959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.422044 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.422046 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:05:51.422101 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 23:05:51.422153 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":22}
I20260810 23:05:51.422279 8128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.418551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:35609 (local address 127.23.56.66:52559)
0810 23:05:51.418756 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.418768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:51.418821 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:51.419026 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:51.419027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:51.419028 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:05:51.419075 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.419076 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.419944 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.419945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.421146 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:51.421150 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.421973 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.421973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:51.422039 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.422043 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:05:51.422101 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 23:05:51.422152 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":35,"thread_start_us":144,"threads_started":1}
I20260810 23:05:51.422657 7749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e714ef4a2bc4ab8841b458d2e987ac0" candidate_uuid: "32a6dc92f3634db384160aa638dc3f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d079a63756c84a569fe91f0fbeaeadd3" is_pre_election: true
I20260810 23:05:51.422685 8055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e714ef4a2bc4ab8841b458d2e987ac0" candidate_uuid: "32a6dc92f3634db384160aa638dc3f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b852721ea34202b33a3942096ebf3e" is_pre_election: true
I20260810 23:05:51.422807 7749 raft_consensus.cc:2454] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a6dc92f3634db384160aa638dc3f9d in term 0.
I20260810 23:05:51.422842 8055 raft_consensus.cc:2454] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32a6dc92f3634db384160aa638dc3f9d in term 0.
I20260810 23:05:51.423094 7830 leader_election.cc:302] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a6dc92f3634db384160aa638dc3f9d, 70b852721ea34202b33a3942096ebf3e; no voters:
I20260810 23:05:51.423250 8125 raft_consensus.cc:2781] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:05:51.423314 8125 raft_consensus.cc:482] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:05:51.423338 8125 raft_consensus.cc:3037] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:51.424034 8125 raft_consensus.cc:504] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.424163 8125 leader_election.cc:288] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [CANDIDATE]: Term 1 election: Requested vote from peers 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095), d079a63756c84a569fe91f0fbeaeadd3 (127.23.56.65:35609)
I20260810 23:05:51.424338 7749 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e714ef4a2bc4ab8841b458d2e987ac0" candidate_uuid: "32a6dc92f3634db384160aa638dc3f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d079a63756c84a569fe91f0fbeaeadd3"
I20260810 23:05:51.424353 8055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e714ef4a2bc4ab8841b458d2e987ac0" candidate_uuid: "32a6dc92f3634db384160aa638dc3f9d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70b852721ea34202b33a3942096ebf3e"
I20260810 23:05:51.424420 7749 raft_consensus.cc:3037] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:51.424438 8055 raft_consensus.cc:3037] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:05:51.425140 7749 raft_consensus.cc:2454] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a6dc92f3634db384160aa638dc3f9d in term 1.
I20260810 23:05:51.425206 8055 raft_consensus.cc:2454] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32a6dc92f3634db384160aa638dc3f9d in term 1.
I20260810 23:05:51.425308 7830 leader_election.cc:302] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32a6dc92f3634db384160aa638dc3f9d, d079a63756c84a569fe91f0fbeaeadd3; no voters:
I20260810 23:05:51.425390 8125 raft_consensus.cc:2781] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:05:51.425551 8125 raft_consensus.cc:686] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 1 LEADER]: Becoming Leader. State: Replica: 32a6dc92f3634db384160aa638dc3f9d, State: Running, Role: LEADER
I20260810 23:05:51.425652 8125 consensus_queue.cc:237] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.426503 7589 catalog_manager.cc:5495] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d reported cstate change: term changed from 0 to 1, leader changed from <none> to 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66). New cstate: current_term: 1 leader_uuid: "32a6dc92f3634db384160aa638dc3f9d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } health_report { overall_health: UNKNOWN } } }
I20260810 23:05:51.487464 8118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.485002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44548 (local address 127.23.56.66:40009)
0810 23:05:51.485072 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:51.485075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:51.485086 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:05:51.485135 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:51.485136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:51.485163 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:51.485243 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:05:51.485411 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.485891 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.486629 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:51.486837 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:51.486918 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:51.487110 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:05:51.487216 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:05:51.487317 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:51.487337 (+ 20us) server_negotiation.cc:299] Negotiation successful
0810 23:05:51.487357 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:05:51.489312 7749 raft_consensus.cc:1261] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Refusing update from remote peer 32a6dc92f3634db384160aa638dc3f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:51.489297 8055 raft_consensus.cc:1261] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [term 1 FOLLOWER]: Refusing update from remote peer 32a6dc92f3634db384160aa638dc3f9d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:05:51.489754 8130 consensus_queue.cc:1035] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:51.489928 8125 consensus_queue.cc:1035] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:05:51.492842 8134 mvcc.cc:205] Tried to move back new op lower bound from 7317107308497367040 to 7317107308240023552. Current Snapshot: MvccSnapshot[applied={T|T < 7317107308497367040}]
I20260810 23:05:51.494206 7589 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:40034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:51.494305 7589 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:40034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:05:51.495164 7589 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Sending DeleteTablet for 3 replicas of tablet 8e714ef4a2bc4ab8841b458d2e987ac0
I20260810 23:05:51.495539 8034 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e714ef4a2bc4ab8841b458d2e987ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:51 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:37958
I20260810 23:05:51.495535 7887 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e714ef4a2bc4ab8841b458d2e987ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:51 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:44532
I20260810 23:05:51.495734 7724 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e714ef4a2bc4ab8841b458d2e987ac0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:05:51 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:43950
I20260810 23:05:51.495864 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 7631
I20260810 23:05:51.495945 8146 tablet_replica.cc:330] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: stopping tablet replica
I20260810 23:05:51.496002 8146 raft_consensus.cc:2229] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:51.496042 8146 raft_consensus.cc:2258] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:51.503479 8145 tablet_replica.cc:330] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: stopping tablet replica
I20260810 23:05:51.503492 8144 tablet_replica.cc:330] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: stopping tablet replica
I20260810 23:05:51.503607 8144 raft_consensus.cc:2229] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:05:51.503705 8145 raft_consensus.cc:2229] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:05:51.503719 8144 raft_consensus.cc:2258] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:51.503898 8145 raft_consensus.cc:2258] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:05:51.504232 8144 ts_tablet_manager.cc:1905] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:05:51.504441 8145 ts_tablet_manager.cc:1905] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:05:51.505802 7573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:35609 (error 108)
I20260810 23:05:51.506361 8144 ts_tablet_manager.cc:1918] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:51.506547 8144 log.cc:1196] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/8e714ef4a2bc4ab8841b458d2e987ac0
I20260810 23:05:51.506552 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.506406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:35609 (local address 127.0.0.1:43962)
0810 23:05:51.506456 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.506492 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:35609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:05:51.506806 8145 ts_tablet_manager.cc:1918] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:05:51.506871 8144 ts_tablet_manager.cc:1939] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 32a6dc92f3634db384160aa638dc3f9d: Deleting consensus metadata
I20260810 23:05:51.506979 8145 log.cc:1196] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/8e714ef4a2bc4ab8841b458d2e987ac0
W20260810 23:05:51.506659 7573 catalog_manager.cc:4584] TS d079a63756c84a569fe91f0fbeaeadd3 (127.23.56.65:35609): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8e714ef4a2bc4ab8841b458d2e987ac0: Network error: Client connection negotiation failed: client connection to 127.23.56.65:35609: connect: Connection refused (error 111)
I20260810 23:05:51.507268 7571 catalog_manager.cc:4841] TS 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009): tablet 8e714ef4a2bc4ab8841b458d2e987ac0 (table test-table [id=441eb506436c405da910faebce30f10f]) successfully deleted
I20260810 23:05:51.507589 8145 ts_tablet_manager.cc:1939] T 8e714ef4a2bc4ab8841b458d2e987ac0 P 70b852721ea34202b33a3942096ebf3e: Deleting consensus metadata
I20260810 23:05:51.507654 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 7806
I20260810 23:05:51.511142 7573 catalog_manager.cc:4841] TS 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095): tablet 8e714ef4a2bc4ab8841b458d2e987ac0 (table test-table [id=441eb506436c405da910faebce30f10f]) successfully deleted
I20260810 23:05:51.514930 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 7964
I20260810 23:05:51.521992 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 7553
I20260810 23:05:51.529273 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38651
--webserver_interface=127.23.56.126
--webserver_port=40961
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:38651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:51.611490 8150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:51.611689 8150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:51.611711 8150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:51.612855 8150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:05:51.612897 8150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:51.612911 8150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:51.612926 8150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:05:51.612977 8150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:05:51.614410 8150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:38651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=40961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8150
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:51.614670 8150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:51.614919 8150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:05:51.619848 8150 server_base.cc:1034] running on GCE node
W20260810 23:05:51.619784 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:51.619783 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:51.619802 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:51.620283 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:51.620579 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:51.621737 8150 hybrid_clock.cc:648] HybridClock initialized: now 1786403151621735 us; error 34 us; skew 500 ppm
Aug 10 23:05:51 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403151, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:51.624269 8150 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:05:51.624737 8150 webserver.cc:460] Webserver started at http://127.23.56.126:40961/ using document root <none> and password file <none>
I20260810 23:05:51.624972 8150 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:51.625042 8150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:51.626395 8150 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:51.627115 8170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:51.627362 8150 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:51.627465 8150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "314daad9919d4a7887cb9397cfa6e6f3"
format_stamp: "Formatted at 2026-08-10 23:05:49 on dist-test-slave-1xrh"
server_key: "08531869c8264b2fc97dbae4a875c376"
server_key_iv: "76b5998b7eb99e02aa944601a1a2301d"
server_key_version: "encryptionkey@0"
I20260810 23:05:51.627803 8150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:51.649483 8150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:51.651552 8150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:51.651710 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:51.656075 8150 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:38651
I20260810 23:05:51.656109 8225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:38651 every 8 connection(s)
I20260810 23:05:51.656502 8150 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:05:51.657996 8226 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:05:51.658654 8226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Bootstrap starting.
I20260810 23:05:51.664659 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 8150
I20260810 23:05:51.664940 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:35609
--local_ip_for_outbound_sockets=127.23.56.65
--tserver_master_addrs=127.23.56.126:38651
--webserver_port=33653
--webserver_interface=127.23.56.65
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:51.667690 8226 log.cc:824] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:51.669914 8226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:05:51.670209 8226 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3: Bootstrap complete.
I20260810 23:05:51.672504 8226 raft_consensus.cc:348] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:51.672744 8226 raft_consensus.cc:729] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 314daad9919d4a7887cb9397cfa6e6f3, State: Initialized, Role: FOLLOWER
I20260810 23:05:51.672928 8226 consensus_queue.cc:260] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:51.673007 8226 raft_consensus.cc:388] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:05:51.673061 8226 raft_consensus.cc:482] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:05:51.673115 8226 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:05:51.674103 8226 raft_consensus.cc:504] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:51.674219 8226 leader_election.cc:302] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 314daad9919d4a7887cb9397cfa6e6f3; no voters:
I20260810 23:05:51.674413 8226 leader_election.cc:288] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:05:51.674448 8230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:05:51.674553 8230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [term 3 LEADER]: Becoming Leader. State: Replica: 314daad9919d4a7887cb9397cfa6e6f3, State: Running, Role: LEADER
I20260810 23:05:51.674638 8230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } }
I20260810 23:05:51.674674 8226 sys_catalog.cc:564] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:05:51.674937 8232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "314daad9919d4a7887cb9397cfa6e6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } } }
I20260810 23:05:51.675203 8232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:51.675104 8231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 314daad9919d4a7887cb9397cfa6e6f3. Latest consensus state: current_term: 3 leader_uuid: "314daad9919d4a7887cb9397cfa6e6f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314daad9919d4a7887cb9397cfa6e6f3" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38651 } } }
I20260810 23:05:51.675468 8231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314daad9919d4a7887cb9397cfa6e6f3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:05:51.675570 8241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:05:51.676373 8241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:05:51.676496 8241 catalog_manager.cc:1259] Loaded cluster ID: e44427ddb0f34cb3b45fdd4ce63a66b7
I20260810 23:05:51.676899 8241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 23:05:51.677187 8241 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f01810bb9ad google::LogMessage::Fail() at ??:0
@ 0x7f01810be88a google::LogMessage::SendToLog() at ??:0
@ 0x7f01810bb4de google::LogMessage::Flush() at ??:0
@ 0x7f01810bca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0188fdc0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f0188fe58b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f0182f890d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0182f7e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f0183de26db start_thread at ??:0
@ 0x7f017fbb961f clone at ??:0
W20260810 23:05:51.759660 8228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:51.759878 8228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:51.759914 8228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:51.761019 8228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:51.761077 8228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:51.761124 8228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:05:51.763190 8228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:35609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=33653
--tserver_master_addrs=127.23.56.126:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:51.763451 8228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:51.763669 8228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:51.768584 8270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:51.768595 8272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:51.768944 8274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:51.769053 8228 server_base.cc:1034] running on GCE node
I20260810 23:05:51.769255 8228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:51.769516 8228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:51.770697 8228 hybrid_clock.cc:648] HybridClock initialized: now 1786403151770683 us; error 28 us; skew 500 ppm
Aug 10 23:05:51 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403151, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:51.773312 8228 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:05:51.773797 8228 webserver.cc:460] Webserver started at http://127.23.56.65:33653/ using document root <none> and password file <none>
I20260810 23:05:51.774036 8228 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:51.774109 8228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:51.775528 8228 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:05:51.776316 8282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:51.776510 8228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:05:51.776665 8228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "d079a63756c84a569fe91f0fbeaeadd3"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "c46e2ca90106b71c332cebc92e9a693a"
server_key_iv: "754947cfa6730b5cd125c553633bef4f"
server_key_version: "encryptionkey@0"
I20260810 23:05:51.777017 8228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:51.811302 8228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:51.813671 8228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:51.813851 8228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:51.814109 8228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:51.814563 8290 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:05:51.816614 8228 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:05:51.816681 8228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 23:05:51.816726 8228 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:05:51.817374 8228 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:05:51.817431 8228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 23:05:51.817515 8290 tablet_bootstrap.cc:492] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Bootstrap starting.
I20260810 23:05:51.824945 8228 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:35609
I20260810 23:05:51.825335 8228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:05:51.827281 8407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:35609 every 8 connection(s)
I20260810 23:05:51.829474 8290 log.cc:824] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:05:51.831935 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 8228
I20260810 23:05:51.832201 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:40009
--local_ip_for_outbound_sockets=127.23.56.66
--tserver_master_addrs=127.23.56.126:38651
--webserver_port=41609
--webserver_interface=127.23.56.66
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:05:51.832269 8290 tablet_bootstrap.cc:492] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:05:51.832584 8290 tablet_bootstrap.cc:492] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Bootstrap complete.
I20260810 23:05:51.833881 8290 ts_tablet_manager.cc:1397] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260810 23:05:51.835319 8290 raft_consensus.cc:348] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.835662 8290 raft_consensus.cc:729] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d079a63756c84a569fe91f0fbeaeadd3, State: Initialized, Role: FOLLOWER
I20260810 23:05:51.835807 8290 consensus_queue.cc:260] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:51.836328 8290 ts_tablet_manager.cc:1428] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260810 23:05:51.889870 8410 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:05:51.826045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:57735)
0810 23:05:51.826234 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.826271 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:51.826711 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:51.889657 (+ 62946us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":105,"threads_started":1}
W20260810 23:05:51.890168 8408 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:38651 (0 consecutive failures): Network error: Failed to ping master at 127.23.56.126:38651: Client connection negotiation failed: client connection to 127.23.56.126:38651: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:05:51.890539 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.890404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:44463)
0810 23:05:51.890458 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.890516 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 23:05:51.890924 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.890837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:57901)
0810 23:05:51.890884 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.890908 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 23:05:51.891048 8408 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:05:51.934295 8412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:51.934492 8412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:51.934514 8412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:51.935739 8412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:51.935784 8412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:51.935811 8412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:05:51.937369 8412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:40009
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=41609
--tserver_master_addrs=127.23.56.126:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:51.937662 8412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:51.937924 8412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:51.942948 8422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:51.942978 8419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:51.943207 8412 server_base.cc:1034] running on GCE node
W20260810 23:05:51.943229 8420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:51.943480 8412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:51.943691 8412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:51.944856 8412 hybrid_clock.cc:648] HybridClock initialized: now 1786403151944831 us; error 41 us; skew 500 ppm
Aug 10 23:05:51 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403151, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:51.947376 8412 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:05:51.947857 8412 webserver.cc:460] Webserver started at http://127.23.56.66:41609/ using document root <none> and password file <none>
I20260810 23:05:51.948099 8412 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:51.948168 8412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:51.949630 8412 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:51.950207 8431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:51.950395 8412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:05:51.950493 8412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "32a6dc92f3634db384160aa638dc3f9d"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "a7173a3e7dea2bc38b8372b7bc5e6033"
server_key_iv: "db2269be9998f160b0238ab056ea54d6"
server_key_version: "encryptionkey@0"
I20260810 23:05:51.950834 8412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:51.966543 8412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:51.968536 8412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:51.968684 8412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:51.968945 8412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:51.969259 8412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:51.969326 8412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:51.969369 8412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:51.969401 8412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:51.975720 8412 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:40009
I20260810 23:05:51.975767 8554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:40009 every 8 connection(s)
I20260810 23:05:51.976117 8412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:05:51.977203 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.976812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:57593)
0810 23:05:51.976977 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.977060 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
W20260810 23:05:51.977726 8555 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:38651 (0 consecutive failures): Network error: Failed to ping master at 127.23.56.126:38651: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
I20260810 23:05:51.978067 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.977956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:38257)
0810 23:05:51.978009 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.978049 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 23:05:51.978397 8559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:51.978320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:54363)
0810 23:05:51.978360 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:51.978383 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 23:05:51.978524 8555 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:05:51.980036 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 8412
I20260810 23:05:51.980325 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:41095
--local_ip_for_outbound_sockets=127.23.56.67
--tserver_master_addrs=127.23.56.126:38651
--webserver_port=34725
--webserver_interface=127.23.56.67
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:05:52.062610 8560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:05:52.062837 8560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:05:52.062875 8560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:05:52.064049 8560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:05:52.064112 8560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:05:52.064159 8560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:05:52.065649 8560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:41095
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=34725
--tserver_master_addrs=127.23.56.126:38651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:05:52.065903 8560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:05:52.066126 8560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:05:52.071173 8565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:52.071182 8568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:05:52.071231 8566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:05:52.071473 8560 server_base.cc:1034] running on GCE node
I20260810 23:05:52.071676 8560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:05:52.071924 8560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:05:52.073091 8560 hybrid_clock.cc:648] HybridClock initialized: now 1786403152073078 us; error 42 us; skew 500 ppm
Aug 10 23:05:52 dist-test-slave-1xrh krb5kdc[6924](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403152, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:05:52.075773 8560 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:05:52.076258 8560 webserver.cc:460] Webserver started at http://127.23.56.67:34725/ using document root <none> and password file <none>
I20260810 23:05:52.076507 8560 fs_manager.cc:359] Metadata directory not provided
I20260810 23:05:52.076586 8560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:05:52.078030 8560 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:05:52.078621 8576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:52.078823 8560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:05:52.078959 8560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "70b852721ea34202b33a3942096ebf3e"
format_stamp: "Formatted at 2026-08-10 23:05:50 on dist-test-slave-1xrh"
server_key: "09a98880d29b27805048d3cb420a5cea"
server_key_iv: "c56ed53c8ab7cb46a4ac4226d4c1d8a8"
server_key_version: "encryptionkey@0"
I20260810 23:05:52.079406 8560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:05:52.097265 8560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:05:52.099331 8560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:05:52.099500 8560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:05:52.099745 8560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:05:52.100071 8560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:05:52.100127 8560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:52.100163 8560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:05:52.100208 8560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:05:52.106309 8560 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:41095
I20260810 23:05:52.106346 8702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:41095 every 8 connection(s)
I20260810 23:05:52.106719 8560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:05:52.107975 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:52.107449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:58563)
0810 23:05:52.107690 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:52.107781 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":79,"threads_started":1}
W20260810 23:05:52.108539 8703 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:38651 (0 consecutive failures): Network error: Failed to ping master at 127.23.56.126:38651: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
I20260810 23:05:52.108842 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:52.108743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:32859)
0810 23:05:52.108789 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:52.108820 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 23:05:52.109112 8705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:52.109038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:43927)
0810 23:05:52.109076 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:52.109097 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 23:05:52.109222 8703 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:05:52.115960 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 8560
I20260810 23:05:52.891779 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:52.891387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:37941)
0810 23:05:52.891563 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:52.891607 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":67,"threads_started":1}
I20260810 23:05:52.979192 8708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:52.978937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:47133)
0810 23:05:52.979056 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:52.979090 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260810 23:05:53.109831 8709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:53.109565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:52509)
0810 23:05:53.109699 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:53.109739 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":57,"threads_started":1}
I20260810 23:05:53.331948 8712 raft_consensus.cc:482] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:53.332118 8712 raft_consensus.cc:504] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:53.332520 8712 leader_election.cc:288] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095), 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
Aug 10 23:05:53 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403151, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.67@KRBTEST.COM
Aug 10 23:05:53 dist-test-slave-1xrh krb5kdc[6924](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403151, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.66@KRBTEST.COM
I20260810 23:05:53.343231 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:53.332716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:40009 (local address 127.23.56.65:56609)
0810 23:05:53.332811 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:53.332824 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:53.332884 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:53.333895 (+ 1011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:53.333899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:53.334300 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:53.334482 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:53.334487 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:53.335682 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:53.335683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:53.336426 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:53.336428 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:53.337045 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:53.337047 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:53.337171 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:53.337833 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:53.337845 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:53.339939 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:53.341771 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:53.341772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:53.341773 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:53.341966 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:53.342227 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:53.342227 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:53.342228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:53.342303 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:53.342685 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:53.342692 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:53.342827 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 23:05:53.342913 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:05:53.343098 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":2,"thread_start_us":56,"threads_started":1}
I20260810 23:05:53.343286 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:53.332854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:44111 (local address 127.23.56.67:41095)
0810 23:05:53.333011 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:53.333026 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:53.333061 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:05:53.333654 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:53.333657 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:53.333701 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:53.333853 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:53.334605 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:53.335555 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:53.336554 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:53.336936 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:53.337070 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:53.339612 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:53.339615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:53.339616 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:53.339626 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:53.341505 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:53.342004 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:53.342008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:53.342008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:53.342108 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:53.342376 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:53.342377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:53.342377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:53.342608 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:53.342702 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:53.343036 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 23:05:53.343181 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":87,"threads_started":1}
I20260810 23:05:53.343281 8715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:53.332859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:56609 (local address 127.23.56.66:40009)
0810 23:05:53.333038 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 23:05:53.333053 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 23:05:53.333081 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:05:53.333683 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:05:53.333686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:05:53.333728 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:05:53.333871 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:05:53.334607 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:53.335554 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:53.336521 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:05:53.336936 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:05:53.337077 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:53.340049 (+ 2972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:05:53.340052 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:05:53.340053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:05:53.340064 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:05:53.341688 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:53.342047 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:53.342049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:53.342050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:53.342129 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:05:53.342385 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:05:53.342386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:05:53.342387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:05:53.342608 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:05:53.342702 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:05:53.342994 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 23:05:53.343166 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":75,"threads_started":1}
I20260810 23:05:53.343343 8707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:53.332684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41095 (local address 127.23.56.65:44111)
0810 23:05:53.332742 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:53.332754 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:05:53.332812 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:05:53.333894 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:05:53.333897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:05:53.334298 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:05:53.334485 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:53.334490 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:53.335659 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:53.335660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:53.336426 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:05:53.336429 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:05:53.337049 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:05:53.337050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:05:53.337168 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:05:53.337834 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:05:53.337851 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:05:53.339485 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:05:53.341591 (+ 2106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:53.341599 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:53.341601 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:53.341910 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:53.342177 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:05:53.342178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:05:53.342178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:05:53.342305 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:05:53.342686 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:05:53.342692 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:05:53.342827 (+ 135us) client_negotiation.cc:769] Sending connection context
0810 23:05:53.342957 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 23:05:53.343204 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":187}
W20260810 23:05:53.344475 8286 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
W20260810 23:05:53.344676 8284 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
I20260810 23:05:53.344722 8284 leader_election.cc:302] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d079a63756c84a569fe91f0fbeaeadd3; no voters: 32a6dc92f3634db384160aa638dc3f9d, 70b852721ea34202b33a3942096ebf3e
I20260810 23:05:53.344875 8712 raft_consensus.cc:2726] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:05:53.892486 8716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:53.892254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:45081)
0810 23:05:53.892401 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:53.892439 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260810 23:05:53.979888 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:53.979669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:38107)
0810 23:05:53.979813 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:53.979847 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":130,"threads_started":1}
I20260810 23:05:54.110540 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:54.110312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:35185)
0810 23:05:54.110455 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:54.110498 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260810 23:05:54.893265 8719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:54.893020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:58373)
0810 23:05:54.893173 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:54.893213 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":80,"threads_started":1}
I20260810 23:05:54.980659 8720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:54.980370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:35523)
0810 23:05:54.980542 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:54.980586 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260810 23:05:55.082171 8721 raft_consensus.cc:482] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:55.082259 8721 raft_consensus.cc:504] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:55.082463 8721 leader_election.cc:288] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095), 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
W20260810 23:05:55.082870 8286 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
W20260810 23:05:55.082952 8284 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
I20260810 23:05:55.083073 8286 leader_election.cc:302] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d079a63756c84a569fe91f0fbeaeadd3; no voters: 32a6dc92f3634db384160aa638dc3f9d, 70b852721ea34202b33a3942096ebf3e
I20260810 23:05:55.083180 8721 raft_consensus.cc:2726] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:05:55.111322 8722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:55.111059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:51027)
0810 23:05:55.111227 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:55.111268 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":94,"threads_started":1}
I20260810 23:05:55.894052 8723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:55.893722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:37043)
0810 23:05:55.893971 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:55.894006 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":88,"threads_started":1}
I20260810 23:05:55.981389 8724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:55.981167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:44837)
0810 23:05:55.981309 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:55.981347 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260810 23:05:56.111999 8725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:56.111785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:51359)
0810 23:05:56.111924 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:56.111957 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260810 23:05:56.860164 8726 raft_consensus.cc:482] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:56.860263 8726 raft_consensus.cc:504] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:56.860448 8726 leader_election.cc:288] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095), 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
W20260810 23:05:56.860809 8286 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
W20260810 23:05:56.860909 8284 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
I20260810 23:05:56.861003 8284 leader_election.cc:302] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d079a63756c84a569fe91f0fbeaeadd3; no voters: 32a6dc92f3634db384160aa638dc3f9d, 70b852721ea34202b33a3942096ebf3e
I20260810 23:05:56.861081 8726 raft_consensus.cc:2726] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:05:56.894845 8727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:56.894589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:54529)
0810 23:05:56.894723 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:56.894763 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260810 23:05:56.982102 8728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:56.981852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:53175)
0810 23:05:56.981995 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:56.982025 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260810 23:05:57.112703 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:57.112476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:33355)
0810 23:05:57.112624 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:57.112660 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260810 23:05:57.895543 8730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:57.895329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:40101)
0810 23:05:57.895459 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:57.895494 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260810 23:05:57.982788 8731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:57.982555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:54511)
0810 23:05:57.982702 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:57.982743 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":63,"threads_started":1}
I20260810 23:05:58.113339 8732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:58.113138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:50359)
0810 23:05:58.113262 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:58.113293 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":62,"threads_started":1}
I20260810 23:05:58.896185 8733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:58.895979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:41943)
0810 23:05:58.896109 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:58.896139 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260810 23:05:58.900305 8734 raft_consensus.cc:482] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:05:58.900374 8734 raft_consensus.cc:504] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:05:58.900576 8734 leader_election.cc:288] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095), 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
W20260810 23:05:58.900980 8284 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
W20260810 23:05:58.901039 8286 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
I20260810 23:05:58.901068 8284 leader_election.cc:302] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d079a63756c84a569fe91f0fbeaeadd3; no voters: 32a6dc92f3634db384160aa638dc3f9d, 70b852721ea34202b33a3942096ebf3e
I20260810 23:05:58.901144 8734 raft_consensus.cc:2726] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:05:58.983572 8735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:58.983314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:60275)
0810 23:05:58.983490 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:58.983525 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260810 23:05:59.114099 8736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:59.113838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:50215)
0810 23:05:59.113985 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:59.114031 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260810 23:05:59.897027 8737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:59.896751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:57389)
0810 23:05:59.896940 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:59.896977 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":84,"threads_started":1}
I20260810 23:05:59.984242 8738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:05:59.984053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:42467)
0810 23:05:59.984167 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 23:05:59.984201 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
I20260810 23:06:00.114866 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:00.114649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:45501)
0810 23:06:00.114781 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:00.114819 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":70,"threads_started":1}
I20260810 23:06:00.897732 8740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:00.897516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:47425)
0810 23:06:00.897637 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:00.897683 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260810 23:06:00.985044 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:00.984818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:41525)
0810 23:06:00.984952 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:00.984996 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":62,"threads_started":1}
I20260810 23:06:01.115556 8742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:01.115358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:45431)
0810 23:06:01.115473 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:01.115508 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":54,"threads_started":1}
I20260810 23:06:01.898504 8743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:01.898267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:48429)
0810 23:06:01.898416 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:01.898449 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260810 23:06:01.985739 8745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:01.985504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:49703)
0810 23:06:01.985649 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:01.985690 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260810 23:06:02.116303 8746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.116052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:44073)
0810 23:06:02.116194 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.116240 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260810 23:06:02.482877 8747 raft_consensus.cc:482] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:02.483022 8747 raft_consensus.cc:504] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:06:02.483273 8747 leader_election.cc:288] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095), 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
W20260810 23:06:02.483824 8286 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
W20260810 23:06:02.483903 8284 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
I20260810 23:06:02.483951 8284 leader_election.cc:302] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d079a63756c84a569fe91f0fbeaeadd3; no voters: 32a6dc92f3634db384160aa638dc3f9d, 70b852721ea34202b33a3942096ebf3e
I20260810 23:06:02.484031 8747 raft_consensus.cc:2726] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:06:02.899302 8748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.899023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:57459)
0810 23:06:02.899205 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.899252 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":84,"threads_started":1}
I20260810 23:06:02.986420 8749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:02.986211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:42895)
0810 23:06:02.986333 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:02.986374 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260810 23:06:03.117038 8750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:03.116792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:45707)
0810 23:06:03.116946 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:03.116992 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":96,"threads_started":1}
I20260810 23:06:03.900115 8751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:03.899883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:59595)
0810 23:06:03.900022 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:03.900061 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260810 23:06:03.987241 8752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:03.986970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:47589)
0810 23:06:03.987104 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:03.987181 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":73,"threads_started":1}
I20260810 23:06:04.117765 8753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:04.117539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:58737)
0810 23:06:04.117685 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:04.117720 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260810 23:06:04.900974 8754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:04.900754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:57995)
0810 23:06:04.900886 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:04.900925 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260810 23:06:04.987895 8755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:04.987688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:38771)
0810 23:06:04.987816 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:04.987851 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260810 23:06:05.118443 8757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.118205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:53009)
0810 23:06:05.118359 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.118397 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260810 23:06:05.901747 8759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.901485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:55075)
0810 23:06:05.901656 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.901694 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":86,"threads_started":1}
I20260810 23:06:05.988508 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:05.988318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:44355)
0810 23:06:05.988433 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:05.988466 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":56,"threads_started":1}
I20260810 23:06:06.119158 8761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.118919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:51855)
0810 23:06:06.119040 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.119074 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
I20260810 23:06:06.819514 8762 raft_consensus.cc:482] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:06.819650 8762 raft_consensus.cc:504] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70b852721ea34202b33a3942096ebf3e" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41095 } } peers { permanent_uuid: "32a6dc92f3634db384160aa638dc3f9d" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 40009 } } peers { permanent_uuid: "d079a63756c84a569fe91f0fbeaeadd3" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35609 } }
I20260810 23:06:06.819880 8762 leader_election.cc:288] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095), 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009)
W20260810 23:06:06.820348 8284 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 32a6dc92f3634db384160aa638dc3f9d (127.23.56.66:40009): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
W20260810 23:06:06.820431 8286 leader_election.cc:341] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 70b852721ea34202b33a3942096ebf3e (127.23.56.67:41095): Not found: Tablet not found: 8e714ef4a2bc4ab8841b458d2e987ac0
I20260810 23:06:06.820475 8286 leader_election.cc:302] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d079a63756c84a569fe91f0fbeaeadd3; no voters: 32a6dc92f3634db384160aa638dc3f9d, 70b852721ea34202b33a3942096ebf3e
I20260810 23:06:06.820565 8762 raft_consensus.cc:2726] T 8e714ef4a2bc4ab8841b458d2e987ac0 P d079a63756c84a569fe91f0fbeaeadd3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:06:06.902530 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.902281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.65:38281)
0810 23:06:06.902448 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.902484 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":75,"threads_started":1}
I20260810 23:06:06.989488 8767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:06.989227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.66:56753)
0810 23:06:06.989375 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:06.989411 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":69,"threads_started":1}
I20260810 23:06:07.119871 8768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:07.119654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38651 (local address 127.23.56.67:43307)
0810 23:06:07.119787 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:07.119821 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:38651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260810 23:06:07.124357 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 8228
I20260810 23:06:07.132302 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 8412
I20260810 23:06:07.137998 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 8560
2026-08-10T23:06:07Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18594 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:06:07 dist-test-slave-1xrh krb5kdc[8772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:06:07 dist-test-slave-1xrh krb5kdc[8772](info): set up 2 sockets
Aug 10 23:06:07 dist-test-slave-1xrh krb5kdc[8772](info): commencing operation
krb5kdc: starting...
W20260810 23:06:08.249914 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.090s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:06:08 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:06:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:06:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:06:08.373852 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34295
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:34295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:08.455523 8787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:08.455758 8787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:08.455806 8787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:08.456941 8787 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:06:08.457005 8787 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:06:08.457026 8787 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:06:08.457053 8787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:08.457087 8787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:08.457118 8787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:08.457149 8787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:08.457180 8787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:08.458626 8787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:34295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:08.458892 8787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:08.459116 8787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:08.464305 8793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:08.464363 8794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:08.464349 8787 server_base.cc:1034] running on GCE node
W20260810 23:06:08.464301 8796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:08.464757 8787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:08.465001 8787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:08.466140 8787 hybrid_clock.cc:648] HybridClock initialized: now 1786403168466117 us; error 37 us; skew 500 ppm
Aug 10 23:06:08 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403168, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:08.468611 8787 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:06:08.469063 8787 webserver.cc:460] Webserver started at http://127.23.56.126:45647/ using document root <none> and password file <none>
I20260810 23:06:08.469276 8787 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:08.469343 8787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:08.469501 8787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:08.470520 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "fd57bf2d6ed04d35a162af020f6f5f86"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "c4a5e16954dd72fc2fddb34e9852c0ce"
server_key_iv: "21918d8104e4b10c3b1ae7b0c6fca2a9"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.470873 8787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "fd57bf2d6ed04d35a162af020f6f5f86"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "c4a5e16954dd72fc2fddb34e9852c0ce"
server_key_iv: "21918d8104e4b10c3b1ae7b0c6fca2a9"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.472245 8787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:06:08.472992 8805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:08.473186 8787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:08.473284 8787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "fd57bf2d6ed04d35a162af020f6f5f86"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "c4a5e16954dd72fc2fddb34e9852c0ce"
server_key_iv: "21918d8104e4b10c3b1ae7b0c6fca2a9"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.473368 8787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:08.480374 8787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:08.482157 8787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:08.482311 8787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:08.486407 8787 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:34295
I20260810 23:06:08.486438 8866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:34295 every 8 connection(s)
I20260810 23:06:08.486770 8787 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:08.487656 8867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:08.488448 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 8787
I20260810 23:06:08.488561 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:06:08.488756 23777 external_mini_cluster.cc:1428] Setting key ee8fcb437ef758d605f79964b278eae4
I20260810 23:06:08.490233 8867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Bootstrap starting.
I20260810 23:06:08.490994 8867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:08.491344 8867 log.cc:824] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:08.492096 8867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: No bootstrap required, opened a new log
I20260810 23:06:08.493983 8867 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:08.494133 8867 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:08.494184 8867 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd57bf2d6ed04d35a162af020f6f5f86, State: Initialized, Role: FOLLOWER
I20260810 23:06:08.494320 8867 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:08.494417 8867 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:08.494458 8867 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:08.494508 8867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:08.495103 8867 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:08.495244 8867 leader_election.cc:302] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd57bf2d6ed04d35a162af020f6f5f86; no voters:
I20260810 23:06:08.495419 8867 leader_election.cc:288] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:06:08.495474 8874 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 1 FOLLOWER]: Leader election won for term 1
Aug 10 23:06:08 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403168, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:06:08.495677 8867 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:08.495685 8874 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 1 LEADER]: Becoming Leader. State: Replica: fd57bf2d6ed04d35a162af020f6f5f86, State: Running, Role: LEADER
I20260810 23:06:08.495784 8874 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:08.496102 8875 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } } }
I20260810 23:06:08.496170 8875 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:08.496352 8876 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd57bf2d6ed04d35a162af020f6f5f86. Latest consensus state: current_term: 1 leader_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } } }
I20260810 23:06:08.496403 8876 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:08.496531 8881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:08.497218 8881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:08.498665 8881 catalog_manager.cc:1347] Generated new cluster ID: 6dfac5861d724782a0eeb8117fb88f5d
I20260810 23:06:08.498715 8881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:08.500021 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:08.489263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36992 (local address 127.23.56.126:34295)
0810 23:06:08.489430 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:08.489437 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:08.489461 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:06:08.489986 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:08.489992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:08.490036 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:08.490142 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:08.490988 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.492103 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.492884 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.493282 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.493398 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:08.496115 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:08.496120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:08.496120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:08.496131 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:08.498331 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:08.498726 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:08.498730 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:08.498730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:08.498833 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:08.499090 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:08.499092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:08.499092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:08.499413 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:08.499569 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:08.499680 (+ 111us) server_negotiation.cc:299] Negotiation successful
0810 23:06:08.499816 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260810 23:06:08.508199 8881 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:06:08.508662 8881 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:08.511612 8881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Generated new TSK 0
I20260810 23:06:08.511796 8881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:06:08.546053 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34295
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:08.644336 8896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:08.644529 8896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:08.644552 8896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:08.645691 8896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:08.645735 8896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:08.645761 8896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:06:08.647224 8896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:08.647476 8896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:08.647714 8896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:08.652736 8902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:08.652709 8904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:08.652707 8906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:08.652808 8896 server_base.cc:1034] running on GCE node
I20260810 23:06:08.653255 8896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:08.653461 8896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:08.654605 8896 hybrid_clock.cc:648] HybridClock initialized: now 1786403168654588 us; error 34 us; skew 500 ppm
Aug 10 23:06:08 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403168, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:08.657110 8896 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:06:08.657553 8896 webserver.cc:460] Webserver started at http://127.23.56.65:36015/ using document root <none> and password file <none>
I20260810 23:06:08.657754 8896 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:08.657826 8896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:08.657923 8896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:08.658871 8896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "92894752a062462482fcde96416d4d38"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "b6b42f08ce689532055409adfa17639c"
server_key_iv: "0f80a81ebd59427f39a3dc92ae7fd226"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.659235 8896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "92894752a062462482fcde96416d4d38"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "b6b42f08ce689532055409adfa17639c"
server_key_iv: "0f80a81ebd59427f39a3dc92ae7fd226"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.660523 8896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:08.661185 8915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:08.661365 8896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:06:08.661449 8896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "92894752a062462482fcde96416d4d38"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "b6b42f08ce689532055409adfa17639c"
server_key_iv: "0f80a81ebd59427f39a3dc92ae7fd226"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.661501 8896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:08.677654 8896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:08.680037 8896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:08.680148 8896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:08.680333 8896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:08.680581 8896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:08.680612 8896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:08.680634 8896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:08.680645 8896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:08.686650 8896 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:40733
I20260810 23:06:08.686707 9055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:40733 every 8 connection(s)
I20260810 23:06:08.687023 8896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:08.691546 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 8896
I20260810 23:06:08.691671 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:06:08.691908 23777 external_mini_cluster.cc:1428] Setting key 9c9e0522e442bf182f7e2387d03d49b6
Aug 10 23:06:08 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403168, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:06:08.696784 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:08.687734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:42627 (local address 127.23.56.126:34295)
0810 23:06:08.687782 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:08.687784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:08.688029 (+ 245us) server_negotiation.cc:407] Connection header received
0810 23:06:08.688555 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:08.688556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:08.688569 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:08.688624 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:08.689555 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.690072 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.691069 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.691340 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.691434 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:08.693769 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:08.693771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:08.693771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:08.693779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:08.695278 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:08.695771 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:08.695772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:08.695772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:08.695825 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:08.696102 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:08.696102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:08.696103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:08.696217 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:08.696266 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:08.696633 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 23:06:08.696719 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:06:08.697021 9058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:08.687718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:42627)
0810 23:06:08.687906 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:08.687942 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:08.688448 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:08.688690 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:08.688694 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:08.689257 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:08.689454 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:08.689459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.690174 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.690175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:08.690964 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:08.690966 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.691449 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.691450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:08.691571 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:08.692129 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:08.692141 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:08.693625 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:08.695358 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:08.695366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:08.695368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:08.695695 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:08.695892 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:08.695893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:08.695894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:08.696036 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:08.696278 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:08.696286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:08.696448 (+ 162us) client_negotiation.cc:769] Sending connection context
0810 23:06:08.696564 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 23:06:08.696742 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":94,"threads_started":1}
I20260810 23:06:08.697710 9056 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34295
I20260810 23:06:08.697825 9056 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:08.698030 9056 heartbeater.cc:511] Master 127.23.56.126:34295 requested a full tablet report, sending...
I20260810 23:06:08.698647 8826 ts_manager.cc:194] Registered new tserver with Master: 92894752a062462482fcde96416d4d38 (127.23.56.65:40733)
I20260810 23:06:08.699642 8826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:42627
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:06:08.718713 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34295
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:08.799806 9062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:08.800004 9062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:08.800028 9062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:08.801080 9062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:08.801113 9062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:08.801138 9062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:06:08.802495 9062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:08.802667 9062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:08.802832 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:06:08.807829 9062 server_base.cc:1034] running on GCE node
W20260810 23:06:08.807727 9071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:08.807750 9069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:08.807736 9068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:08.808235 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:08.808478 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:08.809640 9062 hybrid_clock.cc:648] HybridClock initialized: now 1786403168809623 us; error 35 us; skew 500 ppm
Aug 10 23:06:08 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403168, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:08.812188 9062 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:06:08.812600 9062 webserver.cc:460] Webserver started at http://127.23.56.66:42183/ using document root <none> and password file <none>
I20260810 23:06:08.812844 9062 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:08.812925 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:08.813083 9062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:08.814073 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "eb430ddd29824e619a84a7702896e8cf"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "40da989c97720270ae705056e2d2ebba"
server_key_iv: "5d7d72d6eeeebc9427959a78389bbfa9"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.814431 9062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "eb430ddd29824e619a84a7702896e8cf"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "40da989c97720270ae705056e2d2ebba"
server_key_iv: "5d7d72d6eeeebc9427959a78389bbfa9"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.815951 9062 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:06:08.816763 9079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:08.816943 9062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:08.817040 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "eb430ddd29824e619a84a7702896e8cf"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "40da989c97720270ae705056e2d2ebba"
server_key_iv: "5d7d72d6eeeebc9427959a78389bbfa9"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.817121 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:08.847448 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:08.849491 9062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:08.849637 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:08.849866 9062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:08.850169 9062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:08.850221 9062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:08.850261 9062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:08.850288 9062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:08.856177 9062 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:33751
I20260810 23:06:08.856222 9206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:33751 every 8 connection(s)
I20260810 23:06:08.856567 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
Aug 10 23:06:08 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403168, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:06:08.863670 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 9062
I20260810 23:06:08.863814 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:06:08.864043 23777 external_mini_cluster.cc:1428] Setting key 6af0b2b6bd58285a845a7a7cc8f8c190
I20260810 23:06:08.865998 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:08.857615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:51555 (local address 127.23.56.126:34295)
0810 23:06:08.857679 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:08.857681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:08.857839 (+ 158us) server_negotiation.cc:407] Connection header received
0810 23:06:08.858348 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:08.858349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:08.858363 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:08.858412 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:08.859354 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.859839 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.860784 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.861002 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.861085 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:08.863075 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:08.863076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:08.863077 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:08.863084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:08.864604 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:08.865004 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:08.865005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:08.865005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:08.865063 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:08.865367 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:08.865367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:08.865368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:08.865485 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:08.865560 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:08.865818 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 23:06:08.865899 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:06:08.866173 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:08.857573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:51555)
0810 23:06:08.857728 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:08.857768 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:08.858236 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:08.858474 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:08.858485 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:08.859012 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:08.859262 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:08.859267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.859932 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.859933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:08.860693 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:08.860696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:08.861096 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:08.861097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:08.861223 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:08.861628 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:08.861636 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:08.862969 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:08.864681 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:08.864688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:08.864691 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:08.864907 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:08.865171 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:08.865172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:08.865172 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:08.865276 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:08.865551 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:08.865558 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:08.865689 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 23:06:08.865797 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:06:08.865945 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
I20260810 23:06:08.866740 9207 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34295
I20260810 23:06:08.866823 9207 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:08.867019 9207 heartbeater.cc:511] Master 127.23.56.126:34295 requested a full tablet report, sending...
I20260810 23:06:08.867488 8826 ts_manager.cc:194] Registered new tserver with Master: eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751)
I20260810 23:06:08.867931 8826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:51555
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:06:08.890916 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34295
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:08.970711 9212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:08.970902 9212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:08.970927 9212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:08.972059 9212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:08.972100 9212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:08.972126 9212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:06:08.973531 9212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:34295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:08.973762 9212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:08.973981 9212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:08.979202 9218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:08.979218 9217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:08.979239 9212 server_base.cc:1034] running on GCE node
W20260810 23:06:08.979211 9220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:08.979624 9212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:08.979929 9212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:08.981076 9212 hybrid_clock.cc:648] HybridClock initialized: now 1786403168981057 us; error 38 us; skew 500 ppm
Aug 10 23:06:08 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403168, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:08.983457 9212 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:06:08.983927 9212 webserver.cc:460] Webserver started at http://127.23.56.67:34611/ using document root <none> and password file <none>
I20260810 23:06:08.984153 9212 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:08.984218 9212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:08.984372 9212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:06:08.985441 9212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "5e9262ea11a14e8fb9c19b307914f6dc"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "0b8074a1e2525fed08b48e2838982240"
server_key_iv: "eced417ae5d833041c12532a7344c593"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.985816 9212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "5e9262ea11a14e8fb9c19b307914f6dc"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "0b8074a1e2525fed08b48e2838982240"
server_key_iv: "eced417ae5d833041c12532a7344c593"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.987074 9212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:06:08.987783 9229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:08.987993 9212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:06:08.988097 9212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "5e9262ea11a14e8fb9c19b307914f6dc"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "0b8074a1e2525fed08b48e2838982240"
server_key_iv: "eced417ae5d833041c12532a7344c593"
server_key_version: "encryptionkey@0"
I20260810 23:06:08.988171 9212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:09.002344 9212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:09.004329 9212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:09.004495 9212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:09.004799 9212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:09.005143 9212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:09.005199 9212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:09.005241 9212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:09.005275 9212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:09.011075 9212 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:36521
I20260810 23:06:09.011168 9349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:36521 every 8 connection(s)
I20260810 23:06:09.011551 9212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:09.015332 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 9212
I20260810 23:06:09.015415 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:06:09.015592 23777 external_mini_cluster.cc:1428] Setting key 21aa5e8bc87875c7229ea40212b2086a
Aug 10 23:06:09 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403168, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:06:09.021027 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.012438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:33149 (local address 127.23.56.126:34295)
0810 23:06:09.012496 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:09.012498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:09.012712 (+ 214us) server_negotiation.cc:407] Connection header received
0810 23:06:09.013251 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:09.013252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:09.013261 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:09.013316 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:09.014182 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.014666 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.015982 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.016268 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.016339 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.018383 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:09.018386 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:09.018386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:09.018392 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:09.019875 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:09.020215 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:09.020216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:09.020216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:09.020265 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:09.020463 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:09.020463 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:09.020464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:09.020563 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:09.020612 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:09.020891 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 23:06:09.020963 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 23:06:09.021178 9353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.012426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:33149)
0810 23:06:09.012588 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:09.012630 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:09.013166 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:09.013366 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:09.013369 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:09.013927 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:09.014112 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.014116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.014747 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.014748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.015898 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.015901 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.016373 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.016374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.016520 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.016950 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:09.016957 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:09.018298 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:09.019944 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:09.019950 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:09.019951 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:09.020150 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:09.020321 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:09.020321 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:09.020321 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:09.020412 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:09.020615 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:09.020619 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:09.020763 (+ 144us) client_negotiation.cc:769] Sending connection context
0810 23:06:09.020871 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 23:06:09.020986 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260810 23:06:09.021664 9350 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34295
I20260810 23:06:09.021766 9350 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:09.021950 9350 heartbeater.cc:511] Master 127.23.56.126:34295 requested a full tablet report, sending...
I20260810 23:06:09.022329 8826 ts_manager.cc:194] Registered new tserver with Master: 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521)
I20260810 23:06:09.022691 8826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:33149
I20260810 23:06:09.027257 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:06:09.034117 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.028619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37004 (local address 127.23.56.126:34295)
0810 23:06:09.028688 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:09.028690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:09.028769 (+ 79us) server_negotiation.cc:407] Connection header received
0810 23:06:09.028814 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:09.028814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:09.028823 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:09.028867 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:09.029381 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.029824 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.030543 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.030713 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.030759 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.031997 (+ 1238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:09.031998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:09.031998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:09.032004 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:09.033131 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:09.033433 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:09.033433 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:09.033434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:09.033480 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:09.033606 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:09.033607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:09.033607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:09.033728 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:09.033802 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:09.033940 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 23:06:09.034006 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:06:09.035728 8826 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:09.037230 8826 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:09.043799 9364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.039496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:33751 (local address 127.0.0.1:35180)
0810 23:06:09.039635 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:09.039677 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:09.039747 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:09.040261 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:09.040263 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:09.040266 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:09.040333 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.040335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.041339 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.041340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.042489 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.042491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.043419 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.043420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.043514 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.043530 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 23:06:09.043597 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:06:09.043664 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":65,"threads_started":1}
I20260810 23:06:09.043937 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.039624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35180 (local address 127.23.56.66:33751)
0810 23:06:09.040064 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:09.040069 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:09.040080 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:06:09.040123 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:09.040125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:09.040140 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:09.040417 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:09.040498 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.041225 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.042580 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.043328 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.043404 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.043463 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:09.043764 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 23:06:09.043842 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":99,"threads_started":1}
I20260810 23:06:09.043872 9366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.039707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:40733 (local address 127.0.0.1:53096)
0810 23:06:09.039881 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:09.039889 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:09.039943 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:09.040229 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:09.040232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:09.040235 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:09.040302 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.040304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.041736 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.041737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.042754 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.042756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.043526 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.043527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.043599 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.043604 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:06:09.043667 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 23:06:09.043709 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":136,"threads_started":1}
I20260810 23:06:09.044176 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.040142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.23.56.67:36521)
0810 23:06:09.040306 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:09.040310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:09.040321 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:06:09.040354 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:09.040356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:09.040372 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:09.040445 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:09.040617 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.041672 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.042870 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.043584 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.043665 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.043718 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:09.044090 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 23:06:09.044124 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260810 23:06:09.044266 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.039824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53096 (local address 127.23.56.65:40733)
0810 23:06:09.040030 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:09.040035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:09.040046 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:06:09.040088 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:09.040091 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:09.040108 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:09.040447 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:09.040541 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.041624 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.042837 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.043407 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.043514 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.043573 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:09.044123 (+ 550us) server_negotiation.cc:299] Negotiation successful
0810 23:06:09.044171 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
I20260810 23:06:09.044368 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.039949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:36521 (local address 127.0.0.1:54620)
0810 23:06:09.040049 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:09.040059 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:09.040254 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:09.040504 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:09.040505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:09.040507 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:09.040541 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.040542 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.041775 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.041776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.042780 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.042782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.043677 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.043678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.043745 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.043749 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:06:09.043826 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:06:09.044302 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":69,"threads_started":1}
I20260810 23:06:09.044682 9134 tablet_service.cc:1467] Processing CreateTablet for tablet 1141b78f03fe4bf9a6208c923d159291 (DEFAULT_TABLE table=test-table [id=b72e749f09f04434b919f72f8dbfc69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:09.045070 9134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1141b78f03fe4bf9a6208c923d159291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:09.045050 9280 tablet_service.cc:1467] Processing CreateTablet for tablet 1141b78f03fe4bf9a6208c923d159291 (DEFAULT_TABLE table=test-table [id=b72e749f09f04434b919f72f8dbfc69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:09.045343 9280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1141b78f03fe4bf9a6208c923d159291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:09.047765 9372 tablet_bootstrap.cc:492] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: Bootstrap starting.
I20260810 23:06:09.047861 9373 tablet_bootstrap.cc:492] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: Bootstrap starting.
I20260810 23:06:09.048473 9372 tablet_bootstrap.cc:654] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:09.048540 9373 tablet_bootstrap.cc:654] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:09.048453 8976 tablet_service.cc:1467] Processing CreateTablet for tablet 1141b78f03fe4bf9a6208c923d159291 (DEFAULT_TABLE table=test-table [id=b72e749f09f04434b919f72f8dbfc69b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:09.048771 8976 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1141b78f03fe4bf9a6208c923d159291. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:09.049067 9372 log.cc:824] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:09.049064 9373 log.cc:824] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:09.049964 9373 tablet_bootstrap.cc:492] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: No bootstrap required, opened a new log
I20260810 23:06:09.050040 9373 ts_tablet_manager.cc:1397] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:09.051443 9372 tablet_bootstrap.cc:492] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: No bootstrap required, opened a new log
I20260810 23:06:09.051543 9372 ts_tablet_manager.cc:1397] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 23:06:09.052048 9376 tablet_bootstrap.cc:492] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Bootstrap starting.
I20260810 23:06:09.052141 9373 raft_consensus.cc:348] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:09.052317 9373 raft_consensus.cc:374] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:09.052361 9373 raft_consensus.cc:729] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9262ea11a14e8fb9c19b307914f6dc, State: Initialized, Role: FOLLOWER
I20260810 23:06:09.052464 9373 consensus_queue.cc:260] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:09.052727 9376 tablet_bootstrap.cc:654] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:09.052759 9373 ts_tablet_manager.cc:1428] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:09.052896 9350 heartbeater.cc:503] Master 127.23.56.126:34295 was elected leader, sending a full tablet report...
I20260810 23:06:09.053296 9376 log.cc:824] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:09.053751 9372 raft_consensus.cc:348] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:09.053939 9372 raft_consensus.cc:374] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:09.053989 9372 raft_consensus.cc:729] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb430ddd29824e619a84a7702896e8cf, State: Initialized, Role: FOLLOWER
I20260810 23:06:09.054112 9372 consensus_queue.cc:260] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:09.054158 9376 tablet_bootstrap.cc:492] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: No bootstrap required, opened a new log
I20260810 23:06:09.054217 9376 ts_tablet_manager.cc:1397] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:06:09.054404 9372 ts_tablet_manager.cc:1428] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:09.054425 9207 heartbeater.cc:503] Master 127.23.56.126:34295 was elected leader, sending a full tablet report...
I20260810 23:06:09.055991 9376 raft_consensus.cc:348] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:09.056116 9376 raft_consensus.cc:374] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:09.056137 9376 raft_consensus.cc:729] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92894752a062462482fcde96416d4d38, State: Initialized, Role: FOLLOWER
I20260810 23:06:09.056210 9376 consensus_queue.cc:260] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:09.056401 9376 ts_tablet_manager.cc:1428] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:06:09.056473 9056 heartbeater.cc:503] Master 127.23.56.126:34295 was elected leader, sending a full tablet report...
W20260810 23:06:09.107407 9208 tablet.cc:2369] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:06:09.120653 9379 raft_consensus.cc:482] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:09.120762 9379 raft_consensus.cc:504] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:09.121130 9379 leader_election.cc:288] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92894752a062462482fcde96416d4d38 (127.23.56.65:40733), 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521)
I20260810 23:06:09.124960 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.121512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:57553 (local address 127.23.56.67:36521)
0810 23:06:09.121560 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:09.121562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:09.121694 (+ 132us) server_negotiation.cc:407] Connection header received
0810 23:06:09.121722 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:09.121723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:09.121749 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:09.121877 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:09.122035 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.122717 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.124012 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.124593 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.124672 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.124681 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:09.124847 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:06:09.124890 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:06:09.124960 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.121412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:51413 (local address 127.23.56.65:40733)
0810 23:06:09.121493 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:09.121495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:09.121506 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:06:09.121538 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:09.121538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:09.121551 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:09.121610 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:09.121811 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.122586 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.124018 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.124589 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.124672 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.124681 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:09.124847 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:06:09.124890 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 23:06:09.124967 9209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.121300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:40733 (local address 127.23.56.66:51413)
0810 23:06:09.121363 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:09.121375 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:09.121452 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:09.121625 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:09.121626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:09.121628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:09.121708 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.121710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.122666 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.122667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.123907 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.123910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.124680 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.124681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.124764 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.124766 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:06:09.124823 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 23:06:09.124875 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":7}
I20260810 23:06:09.124974 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.121425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:36521 (local address 127.23.56.66:57553)
0810 23:06:09.121614 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:09.121624 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:09.121680 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:09.121899 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:09.121900 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:09.121902 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:09.121956 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.121957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.122784 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.122785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.123907 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:09.123910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.124680 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.124681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:09.124761 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.124765 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:06:09.124823 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 23:06:09.124874 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":8,"thread_start_us":132,"threads_started":1}
I20260810 23:06:09.125592 9304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1141b78f03fe4bf9a6208c923d159291" candidate_uuid: "eb430ddd29824e619a84a7702896e8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" is_pre_election: true
I20260810 23:06:09.125644 9003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1141b78f03fe4bf9a6208c923d159291" candidate_uuid: "eb430ddd29824e619a84a7702896e8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92894752a062462482fcde96416d4d38" is_pre_election: true
I20260810 23:06:09.125751 9304 raft_consensus.cc:2454] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb430ddd29824e619a84a7702896e8cf in term 0.
I20260810 23:06:09.125829 9003 raft_consensus.cc:2454] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb430ddd29824e619a84a7702896e8cf in term 0.
I20260810 23:06:09.126012 9082 leader_election.cc:302] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9262ea11a14e8fb9c19b307914f6dc, eb430ddd29824e619a84a7702896e8cf; no voters:
I20260810 23:06:09.126161 9379 raft_consensus.cc:2781] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:09.126230 9379 raft_consensus.cc:482] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:09.126248 9379 raft_consensus.cc:3037] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:09.126972 9379 raft_consensus.cc:504] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:09.127187 9379 leader_election.cc:288] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [CANDIDATE]: Term 1 election: Requested vote from peers 92894752a062462482fcde96416d4d38 (127.23.56.65:40733), 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521)
I20260810 23:06:09.127357 9003 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1141b78f03fe4bf9a6208c923d159291" candidate_uuid: "eb430ddd29824e619a84a7702896e8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92894752a062462482fcde96416d4d38"
I20260810 23:06:09.127426 9304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1141b78f03fe4bf9a6208c923d159291" candidate_uuid: "eb430ddd29824e619a84a7702896e8cf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9262ea11a14e8fb9c19b307914f6dc"
I20260810 23:06:09.127445 9003 raft_consensus.cc:3037] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:09.127516 9304 raft_consensus.cc:3037] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:09.128151 9003 raft_consensus.cc:2454] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb430ddd29824e619a84a7702896e8cf in term 1.
I20260810 23:06:09.128330 9304 raft_consensus.cc:2454] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb430ddd29824e619a84a7702896e8cf in term 1.
I20260810 23:06:09.128407 9082 leader_election.cc:302] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92894752a062462482fcde96416d4d38, eb430ddd29824e619a84a7702896e8cf; no voters:
I20260810 23:06:09.128504 9379 raft_consensus.cc:2781] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:09.128700 9379 raft_consensus.cc:686] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 1 LEADER]: Becoming Leader. State: Replica: eb430ddd29824e619a84a7702896e8cf, State: Running, Role: LEADER
I20260810 23:06:09.128794 9379 consensus_queue.cc:237] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:09.129585 8826 catalog_manager.cc:5495] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf reported cstate change: term changed from 0 to 1, leader changed from <none> to eb430ddd29824e619a84a7702896e8cf (127.23.56.66). New cstate: current_term: 1 leader_uuid: "eb430ddd29824e619a84a7702896e8cf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } health_report { overall_health: UNKNOWN } } }
I20260810 23:06:09.134523 9365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.132183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.23.56.66:33751)
0810 23:06:09.132227 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:09.132230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:09.132239 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:06:09.132311 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:09.132312 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:09.132330 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:09.132413 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:09.132555 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.132999 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.133730 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:09.133933 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:09.134006 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:09.134186 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:09.134290 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:09.134382 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:09.134403 (+ 21us) server_negotiation.cc:299] Negotiation successful
0810 23:06:09.134423 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:06:09.136142 9003 raft_consensus.cc:1261] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Refusing update from remote peer eb430ddd29824e619a84a7702896e8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:09.136171 9304 raft_consensus.cc:1261] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [term 1 FOLLOWER]: Refusing update from remote peer eb430ddd29824e619a84a7702896e8cf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:09.136468 9379 consensus_queue.cc:1035] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:09.136591 9382 consensus_queue.cc:1035] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:09.138659 9385 mvcc.cc:205] Tried to move back new op lower bound from 7317107380779220992 to 7317107380752039936. Current Snapshot: MvccSnapshot[applied={T|T < 7317107380779220992}]
I20260810 23:06:09.139292 9386 mvcc.cc:205] Tried to move back new op lower bound from 7317107380779220992 to 7317107380752039936. Current Snapshot: MvccSnapshot[applied={T|T < 7317107380779220992}]
I20260810 23:06:09.141350 8825 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:09.141425 8825 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:09.142109 8825 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Sending DeleteTablet for 3 replicas of tablet 1141b78f03fe4bf9a6208c923d159291
I20260810 23:06:09.142393 9134 tablet_service.cc:1514] Processing DeleteTablet for tablet 1141b78f03fe4bf9a6208c923d159291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:09 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:35180
I20260810 23:06:09.142515 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 8896
I20260810 23:06:09.142555 9280 tablet_service.cc:1514] Processing DeleteTablet for tablet 1141b78f03fe4bf9a6208c923d159291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:09 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:54620
I20260810 23:06:09.142628 8976 tablet_service.cc:1514] Processing DeleteTablet for tablet 1141b78f03fe4bf9a6208c923d159291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:09 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:53096
I20260810 23:06:09.159271 9398 tablet_replica.cc:330] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: stopping tablet replica
I20260810 23:06:09.159377 9398 raft_consensus.cc:2229] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:09.159462 9398 raft_consensus.cc:2258] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:09.159631 9397 tablet_replica.cc:330] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: stopping tablet replica
I20260810 23:06:09.159734 9397 raft_consensus.cc:2229] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:09.159861 9397 raft_consensus.cc:2258] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:09.159960 9398 ts_tablet_manager.cc:1905] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:09.160279 9397 ts_tablet_manager.cc:1905] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 23:06:09.161958 8807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:40733 (error 108)
I20260810 23:06:09.162065 9397 ts_tablet_manager.cc:1918] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:09.162145 9397 log.cc:1196] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/1141b78f03fe4bf9a6208c923d159291
I20260810 23:06:09.162295 9398 ts_tablet_manager.cc:1918] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:09.162431 9397 ts_tablet_manager.cc:1939] T 1141b78f03fe4bf9a6208c923d159291 P eb430ddd29824e619a84a7702896e8cf: Deleting consensus metadata
I20260810 23:06:09.162438 9398 log.cc:1196] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/1141b78f03fe4bf9a6208c923d159291
I20260810 23:06:09.162791 8807 catalog_manager.cc:4841] TS eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751): tablet 1141b78f03fe4bf9a6208c923d159291 (table test-table [id=b72e749f09f04434b919f72f8dbfc69b]) successfully deleted
I20260810 23:06:09.162827 9398 ts_tablet_manager.cc:1939] T 1141b78f03fe4bf9a6208c923d159291 P 5e9262ea11a14e8fb9c19b307914f6dc: Deleting consensus metadata
I20260810 23:06:09.163106 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:09.162964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:40733 (local address 127.0.0.1:53102)
0810 23:06:09.163035 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:09.163080 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:40733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 23:06:09.163272 8807 catalog_manager.cc:4584] TS 92894752a062462482fcde96416d4d38 (127.23.56.65:40733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1141b78f03fe4bf9a6208c923d159291: Network error: Client connection negotiation failed: client connection to 127.23.56.65:40733: connect: Connection refused (error 111)
I20260810 23:06:09.163374 8807 catalog_manager.cc:4841] TS 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521): tablet 1141b78f03fe4bf9a6208c923d159291 (table test-table [id=b72e749f09f04434b919f72f8dbfc69b]) successfully deleted
I20260810 23:06:09.163401 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 9062
I20260810 23:06:09.170025 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 9212
I20260810 23:06:09.175285 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 8787
I20260810 23:06:29.181723 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34295
--webserver_interface=127.23.56.126
--webserver_port=45647
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:34295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:29.264787 9400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:29.264993 9400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:29.265015 9400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:29.266151 9400 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:06:29.266191 9400 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:06:29.266204 9400 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:06:29.266216 9400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:29.266268 9400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:29.266286 9400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:29.266310 9400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:29.266335 9400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:29.267839 9400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:34295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=45647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:29.268077 9400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:29.268335 9400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:29.277338 9409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:29.277338 9407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:29.277400 9406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:29.277680 9400 server_base.cc:1034] running on GCE node
I20260810 23:06:29.277959 9400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:29.278265 9400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:29.279434 9400 hybrid_clock.cc:648] HybridClock initialized: now 1786403189279417 us; error 44 us; skew 500 ppm
Aug 10 23:06:29 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403189, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:29.281915 9400 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:06:29.282343 9400 webserver.cc:460] Webserver started at http://127.23.56.126:45647/ using document root <none> and password file <none>
I20260810 23:06:29.282572 9400 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:29.282653 9400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:29.284122 9400 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:29.284767 9416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.284940 9400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:06:29.285054 9400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "fd57bf2d6ed04d35a162af020f6f5f86"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "c4a5e16954dd72fc2fddb34e9852c0ce"
server_key_iv: "21918d8104e4b10c3b1ae7b0c6fca2a9"
server_key_version: "encryptionkey@0"
I20260810 23:06:29.285437 9400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:29.310900 9400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:29.313938 9400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:29.314139 9400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:29.319531 9400 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:34295
I20260810 23:06:29.319585 9477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:34295 every 8 connection(s)
I20260810 23:06:29.319981 9400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:29.321739 9478 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:06:29.322433 9478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Bootstrap starting.
I20260810 23:06:29.327561 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 9400
I20260810 23:06:29.327833 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:40733
--local_ip_for_outbound_sockets=127.23.56.65
--tserver_master_addrs=127.23.56.126:34295
--webserver_port=36015
--webserver_interface=127.23.56.65
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:29.331804 9478 log.cc:824] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:29.334439 9478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:06:29.334785 9478 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Bootstrap complete.
I20260810 23:06:29.338879 9478 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:29.339280 9478 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd57bf2d6ed04d35a162af020f6f5f86, State: Initialized, Role: FOLLOWER
I20260810 23:06:29.339499 9478 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:29.339588 9478 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:29.339653 9478 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:29.339711 9478 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 1 FOLLOWER]: Advancing to term 2
I20260810 23:06:29.341218 9478 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:29.341337 9478 leader_election.cc:302] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd57bf2d6ed04d35a162af020f6f5f86; no voters:
I20260810 23:06:29.341490 9478 leader_election.cc:288] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 23:06:29.341566 9482 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 23:06:29.341727 9478 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:29.341768 9482 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 2 LEADER]: Becoming Leader. State: Replica: fd57bf2d6ed04d35a162af020f6f5f86, State: Running, Role: LEADER
I20260810 23:06:29.343151 9482 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:29.343711 9484 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd57bf2d6ed04d35a162af020f6f5f86. Latest consensus state: current_term: 2 leader_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } } }
I20260810 23:06:29.343921 9484 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:29.343689 9483 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } } }
I20260810 23:06:29.344058 9483 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:29.344287 9496 catalog_manager.cc:1259] Loaded cluster ID: 6dfac5861d724782a0eeb8117fb88f5d
I20260810 23:06:29.344344 9496 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: loading cluster ID for follower catalog manager: success
I20260810 23:06:29.345561 9496 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: acquiring CA information for follower catalog manager: success
I20260810 23:06:29.345937 9496 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:06:29.346005 9499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:29.346374 9499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:29.346446 9499 catalog_manager.cc:1259] Loaded cluster ID: 6dfac5861d724782a0eeb8117fb88f5d
I20260810 23:06:29.346474 9499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:29.346624 9499 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:29.346692 9499 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Loaded TSK: 0
I20260810 23:06:29.355782 9499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Generated new TSK 1
I20260810 23:06:29.355867 9499 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 23:06:29.419330 9480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:29.419533 9480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:29.419555 9480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:29.420735 9480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:29.420780 9480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:29.420807 9480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:06:29.422901 9480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:40733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=36015
--tserver_master_addrs=127.23.56.126:34295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:29.423301 9480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:29.423607 9480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:29.429090 9511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:29.429090 9507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:29.429225 9508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:29.429826 9480 server_base.cc:1034] running on GCE node
I20260810 23:06:29.430068 9480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:29.430328 9480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:29.431517 9480 hybrid_clock.cc:648] HybridClock initialized: now 1786403189431506 us; error 31 us; skew 500 ppm
Aug 10 23:06:29 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403189, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:29.434098 9480 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:06:29.434525 9480 webserver.cc:460] Webserver started at http://127.23.56.65:36015/ using document root <none> and password file <none>
I20260810 23:06:29.434777 9480 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:29.434861 9480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:29.436311 9480 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:29.437099 9518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.437286 9480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:29.437390 9480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "92894752a062462482fcde96416d4d38"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "b6b42f08ce689532055409adfa17639c"
server_key_iv: "0f80a81ebd59427f39a3dc92ae7fd226"
server_key_version: "encryptionkey@0"
I20260810 23:06:29.437753 9480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:29.458983 9480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:29.461289 9480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:29.461472 9480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:29.461743 9480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:29.462253 9525 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:06:29.463809 9480 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:06:29.463865 9480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 23:06:29.463891 9480 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:06:29.464545 9480 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:06:29.464591 9480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 23:06:29.464632 9525 tablet_bootstrap.cc:492] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Bootstrap starting.
I20260810 23:06:29.472366 9480 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:40733
I20260810 23:06:29.472854 9480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:29.475304 9641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:40733 every 8 connection(s)
I20260810 23:06:29.475392 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 9480
I20260810 23:06:29.475732 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:33751
--local_ip_for_outbound_sockets=127.23.56.66
--tserver_master_addrs=127.23.56.126:34295
--webserver_port=42183
--webserver_interface=127.23.56.66
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:29.476125 9525 log.cc:824] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:29.478609 9525 tablet_bootstrap.cc:492] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:06:29.479001 9525 tablet_bootstrap.cc:492] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Bootstrap complete.
I20260810 23:06:29.480283 9525 ts_tablet_manager.cc:1397] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.009s
I20260810 23:06:29.481254 9525 raft_consensus.cc:348] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:29.481566 9525 raft_consensus.cc:729] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92894752a062462482fcde96416d4d38, State: Initialized, Role: FOLLOWER
I20260810 23:06:29.481721 9525 consensus_queue.cc:260] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } } peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } }
I20260810 23:06:29.482005 9525 ts_tablet_manager.cc:1428] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Aug 10 23:06:29 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403189, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:06:29.490252 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.473563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:45247)
0810 23:06:29.477685 (+ 4122us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:29.477734 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:29.478706 (+ 972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:29.479331 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:29.479335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:29.479731 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:29.479998 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.480004 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.481914 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.481915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.483551 (+ 1636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.483555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.484129 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.484131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.484307 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.484878 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:29.484890 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:29.486548 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:29.488599 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:29.488606 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:29.488608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:29.488825 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:29.489092 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:29.489092 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:29.489093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:29.489257 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:29.489634 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:29.489641 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:29.489775 (+ 134us) client_negotiation.cc:769] Sending connection context
0810 23:06:29.489877 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:06:29.490058 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4069,"thread_start_us":114,"threads_started":1}
I20260810 23:06:29.490252 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.473850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:45247 (local address 127.23.56.126:34295)
0810 23:06:29.474049 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.474061 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.477858 (+ 3797us) server_negotiation.cc:407] Connection header received
0810 23:06:29.478957 (+ 1099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.478965 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.479030 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.479192 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:29.480138 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.481767 (+ 1629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.483647 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.484027 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.484111 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.486655 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:29.486659 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:29.486660 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:29.486685 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:29.488515 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.488912 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.488915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.488915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.489013 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.489334 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.489335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.489335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.489560 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:29.489656 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.489943 (+ 287us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.490070 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":81,"threads_started":1}
I20260810 23:06:29.491088 9642 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34295
I20260810 23:06:29.491230 9642 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:29.491436 9642 heartbeater.cc:511] Master 127.23.56.126:34295 requested a full tablet report, sending...
I20260810 23:06:29.492261 9435 ts_manager.cc:194] Registered new tserver with Master: 92894752a062462482fcde96416d4d38 (127.23.56.65:40733)
I20260810 23:06:29.493638 9435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:45247
I20260810 23:06:29.494633 9642 heartbeater.cc:503] Master 127.23.56.126:34295 was elected leader, sending a full tablet report...
Aug 10 23:06:29 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403189, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.65@KRBTEST.COM
I20260810 23:06:29.501205 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.492897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:40733 (local address 127.0.0.1:60538)
0810 23:06:29.493077 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:29.493117 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:29.493171 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:29.494252 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:29.494258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:29.494512 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:29.494557 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.494559 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.495534 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.495535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.496084 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.496086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.496531 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.496532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.496609 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.497115 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:29.497119 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:29.498324 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:29.500049 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:29.500056 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:29.500058 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:29.500251 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:29.500469 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:29.500470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:29.500471 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:29.500540 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:29.500811 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:29.500818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:29.500868 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 23:06:29.500964 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:06:29.501080 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":119,"threads_started":1}
I20260810 23:06:29.501204 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.493937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60538 (local address 127.23.56.65:40733)
0810 23:06:29.494112 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.494118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.494131 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:06:29.494157 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.494158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.494169 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.494249 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:29.494637 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.495442 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.496178 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.496451 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.496531 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.498411 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:29.498419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:29.498419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:29.498428 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:29.499939 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.500342 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.500346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.500346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.500403 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.500604 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.500605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.500605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.500739 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:29.500830 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.501021 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.501110 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":99,"threads_started":1}
I20260810 23:06:29.501793 9572 tablet_service.cc:1514] Processing DeleteTablet for tablet 1141b78f03fe4bf9a6208c923d159291 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:09 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:60538
I20260810 23:06:29.502319 9652 tablet_replica.cc:330] stopping tablet replica
I20260810 23:06:29.502439 9652 raft_consensus.cc:2229] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:29.502513 9652 raft_consensus.cc:2258] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:29.502871 9652 ts_tablet_manager.cc:1905] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:29.505718 9652 ts_tablet_manager.cc:1918] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:29.505808 9652 log.cc:1196] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal/wals/1141b78f03fe4bf9a6208c923d159291
I20260810 23:06:29.506067 9652 ts_tablet_manager.cc:1939] T 1141b78f03fe4bf9a6208c923d159291 P 92894752a062462482fcde96416d4d38: Deleting consensus metadata
I20260810 23:06:29.506601 9419 catalog_manager.cc:4841] TS 92894752a062462482fcde96416d4d38 (127.23.56.65:40733): tablet 1141b78f03fe4bf9a6208c923d159291 (table test-table [id=b72e749f09f04434b919f72f8dbfc69b]) successfully deleted
W20260810 23:06:29.570868 9648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:29.571080 9648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:29.571105 9648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:29.572352 9648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:29.572396 9648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:29.572424 9648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:06:29.573910 9648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:33751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=42183
--tserver_master_addrs=127.23.56.126:34295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:29.574148 9648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:29.574385 9648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:29.579381 9659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:29.579432 9658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:29.579387 9661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:29.579612 9648 server_base.cc:1034] running on GCE node
I20260810 23:06:29.580777 9648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:29.581022 9648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:29.582180 9648 hybrid_clock.cc:648] HybridClock initialized: now 1786403189582169 us; error 43 us; skew 500 ppm
Aug 10 23:06:29 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403189, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:29.584601 9648 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:06:29.585016 9648 webserver.cc:460] Webserver started at http://127.23.56.66:42183/ using document root <none> and password file <none>
I20260810 23:06:29.585230 9648 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:29.585310 9648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:29.586635 9648 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:29.587242 9668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.587407 9648 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.587503 9648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "eb430ddd29824e619a84a7702896e8cf"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "40da989c97720270ae705056e2d2ebba"
server_key_iv: "5d7d72d6eeeebc9427959a78389bbfa9"
server_key_version: "encryptionkey@0"
I20260810 23:06:29.587838 9648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:29.605387 9648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:29.607189 9648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:29.607337 9648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:29.607565 9648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:29.607872 9648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:29.607930 9648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.607971 9648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:29.608006 9648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.614176 9648 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:33751
I20260810 23:06:29.614233 9792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:33751 every 8 connection(s)
I20260810 23:06:29.614565 9648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
Aug 10 23:06:29 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403189, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:06:29.623317 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.615357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:52261 (local address 127.23.56.126:34295)
0810 23:06:29.615426 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.615428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.615621 (+ 193us) server_negotiation.cc:407] Connection header received
0810 23:06:29.616098 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.616099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.616113 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.616167 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:29.617104 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.617595 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.618508 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.618707 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.618759 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.620894 (+ 2135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:29.620896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:29.620896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:29.620903 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:29.622111 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.622460 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.622461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.622461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.622511 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.622716 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.622716 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.622716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.622828 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:29.622881 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.623111 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.623240 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:06:29.623337 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 9648
I20260810 23:06:29.623522 9795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.615345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:52261)
0810 23:06:29.615517 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:29.615553 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:29.616007 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:29.616244 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:29.616248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:29.616832 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:29.617022 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.617026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.617671 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.617672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.618431 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.618432 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.618778 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.618779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.618873 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.619414 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:29.619422 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:29.620792 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:29.622185 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:29.622192 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:29.622194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:29.622396 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:29.622570 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:29.622571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:29.622572 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:29.622661 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:29.622886 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:29.622890 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:29.623003 (+ 113us) client_negotiation.cc:769] Sending connection context
0810 23:06:29.623082 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:06:29.623280 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260810 23:06:29.623647 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:36521
--local_ip_for_outbound_sockets=127.23.56.67
--tserver_master_addrs=127.23.56.126:34295
--webserver_port=34611
--webserver_interface=127.23.56.67
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:29.624002 9793 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34295
I20260810 23:06:29.624110 9793 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:29.624285 9793 heartbeater.cc:511] Master 127.23.56.126:34295 requested a full tablet report, sending...
I20260810 23:06:29.624723 9435 ts_manager.cc:194] Registered new tserver with Master: eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751)
I20260810 23:06:29.625113 9435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:52261
W20260810 23:06:29.706275 9796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:29.706504 9796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:29.706532 9796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:29.707643 9796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:29.707679 9796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:29.707705 9796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:06:29.709071 9796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:36521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=34611
--tserver_master_addrs=127.23.56.126:34295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:29.709296 9796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:29.709604 9796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:29.714776 9803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:29.714782 9802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:29.714782 9805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:29.714880 9796 server_base.cc:1034] running on GCE node
I20260810 23:06:29.715288 9796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:29.715545 9796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:29.716691 9796 hybrid_clock.cc:648] HybridClock initialized: now 1786403189716671 us; error 38 us; skew 500 ppm
Aug 10 23:06:29 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403189, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:29.719211 9796 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:06:29.719656 9796 webserver.cc:460] Webserver started at http://127.23.56.67:34611/ using document root <none> and password file <none>
I20260810 23:06:29.719875 9796 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:29.719952 9796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:29.721218 9796 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:29.721889 9813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.722060 9796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.722152 9796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "5e9262ea11a14e8fb9c19b307914f6dc"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "0b8074a1e2525fed08b48e2838982240"
server_key_iv: "eced417ae5d833041c12532a7344c593"
server_key_version: "encryptionkey@0"
I20260810 23:06:29.722461 9796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:29.740312 9796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:29.742739 9796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:29.742892 9796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:29.743158 9796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:29.743487 9796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:29.743538 9796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.743582 9796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:29.743609 9796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:29.749131 9796 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:36521
I20260810 23:06:29.749194 9940 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:36521 every 8 connection(s)
I20260810 23:06:29.749559 9796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
Aug 10 23:06:29 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403189, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:06:29.758052 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.750265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:59165 (local address 127.23.56.126:34295)
0810 23:06:29.750334 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.750336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.750516 (+ 180us) server_negotiation.cc:407] Connection header received
0810 23:06:29.750981 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.750982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.750991 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.751051 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:29.751984 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.752459 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.753372 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.753563 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.753613 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.755682 (+ 2069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:29.755684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:29.755684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:29.755691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:29.756977 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.757319 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.757320 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.757320 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.757368 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.757573 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.757573 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.757574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.757668 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:29.757718 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.757932 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.757991 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:06:29.758255 9943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.750226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:59165)
0810 23:06:29.750378 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:29.750420 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:29.750897 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:29.751103 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:29.751106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:29.751700 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:29.751900 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.751905 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.752537 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.752538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.753294 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.753296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.753633 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.753633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.753726 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.754202 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:29.754210 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:29.755598 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:29.757046 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:29.757052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:29.757054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:29.757253 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:29.757423 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:29.757423 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:29.757423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:29.757519 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:29.757719 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:29.757723 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:29.757824 (+ 101us) client_negotiation.cc:769] Sending connection context
0810 23:06:29.757901 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:06:29.758019 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260810 23:06:29.758766 9941 heartbeater.cc:347] Connected to a master server at 127.23.56.126:34295
I20260810 23:06:29.758859 9941 heartbeater.cc:464] Registering TS with master...
I20260810 23:06:29.759003 9941 heartbeater.cc:511] Master 127.23.56.126:34295 requested a full tablet report, sending...
I20260810 23:06:29.759378 9435 ts_manager.cc:194] Registered new tserver with Master: 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521)
I20260810 23:06:29.759665 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 9796
I20260810 23:06:29.759871 9435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:59165
I20260810 23:06:29.765403 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.760038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.23.56.126:34295)
0810 23:06:29.760080 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.760082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.760265 (+ 183us) server_negotiation.cc:407] Connection header received
0810 23:06:29.760313 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.760314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.760324 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.760388 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:29.760918 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.761336 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.762119 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.762281 (+ 162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.762328 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.763422 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:29.763424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:29.763424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:29.763430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:29.764552 (+ 1122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.764840 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.764840 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.764841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.764886 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:29.765046 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:29.765047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:29.765047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:29.765132 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:29.765254 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.765299 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.765349 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:06:29.766180 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:06:29.768641 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.766521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44106 (local address 127.23.56.126:34295)
0810 23:06:29.766554 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.766556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.766619 (+ 63us) server_negotiation.cc:407] Connection header received
0810 23:06:29.766673 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.766674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.766699 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.766757 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:29.766827 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.767332 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.768048 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.768218 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.768265 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.768439 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:29.768523 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:29.768574 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.768589 (+ 15us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.768601 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:06:29.768939 9435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:44106:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:06:29.770622 9435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:06:29.773221 9572 tablet_service.cc:1467] Processing CreateTablet for tablet 81e9181927ae493b8cfa9e66808e811b (DEFAULT_TABLE table=test-table [id=96319cead07441209954f5c2e66e1c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:29.773450 9572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e9181927ae493b8cfa9e66808e811b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:29.774967 9525 tablet_bootstrap.cc:492] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Bootstrap starting.
I20260810 23:06:29.775322 9525 tablet_bootstrap.cc:654] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:29.775812 9525 tablet_bootstrap.cc:492] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: No bootstrap required, opened a new log
I20260810 23:06:29.775882 9525 ts_tablet_manager.cc:1397] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:29.776216 9525 raft_consensus.cc:348] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:29.776298 9525 raft_consensus.cc:374] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:29.776342 9525 raft_consensus.cc:729] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92894752a062462482fcde96416d4d38, State: Initialized, Role: FOLLOWER
I20260810 23:06:29.776394 9525 consensus_queue.cc:260] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:29.776485 9525 ts_tablet_manager.cc:1428] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:29.776866 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.772864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55600 (local address 127.23.56.66:33751)
0810 23:06:29.773057 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.773062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.773072 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:06:29.773115 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.773117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.773132 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.773219 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:29.773458 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.774479 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.775750 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.776380 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.776429 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.776488 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.776749 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.776797 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":126,"threads_started":1}
I20260810 23:06:29.776857 9650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.772483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:36521 (local address 127.0.0.1:58070)
0810 23:06:29.772536 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:29.772565 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:29.772616 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:29.773053 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:29.773054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:29.773055 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:29.773104 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.773106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.774245 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.774246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.775399 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.775402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.776572 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.776573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.776647 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.776670 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:06:29.776727 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 23:06:29.776762 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":7}
I20260810 23:06:29.777094 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.772621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.23.56.67:36521)
0810 23:06:29.772890 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.772894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.772902 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:06:29.772932 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.772933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.772947 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.773048 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:29.773162 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.774156 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.775504 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.776466 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.776556 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.776613 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.776990 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.777032 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260810 23:06:29.776857 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.772586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:33751 (local address 127.0.0.1:55600)
0810 23:06:29.772700 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:29.772711 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:29.772761 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:29.773227 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:29.773228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:29.773230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:29.773274 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.773275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.774573 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.774574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.775653 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.775655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.776575 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.776576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.776644 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.776667 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 23:06:29.776727 (+ 60us) client_negotiation.cc:240] Negotiation successful
0810 23:06:29.776761 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260810 23:06:29.777911 9871 tablet_service.cc:1467] Processing CreateTablet for tablet 81e9181927ae493b8cfa9e66808e811b (DEFAULT_TABLE table=test-table [id=96319cead07441209954f5c2e66e1c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:29.777907 9725 tablet_service.cc:1467] Processing CreateTablet for tablet 81e9181927ae493b8cfa9e66808e811b (DEFAULT_TABLE table=test-table [id=96319cead07441209954f5c2e66e1c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:06:29.778259 9871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e9181927ae493b8cfa9e66808e811b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:29.778259 9725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81e9181927ae493b8cfa9e66808e811b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:06:29.781788 9950 tablet_bootstrap.cc:492] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: Bootstrap starting.
I20260810 23:06:29.782029 9951 tablet_bootstrap.cc:492] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: Bootstrap starting.
I20260810 23:06:29.782660 9951 tablet_bootstrap.cc:654] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:29.783110 9951 log.cc:824] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:29.783869 9950 tablet_bootstrap.cc:654] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: Neither blocks nor log segments found. Creating new log.
I20260810 23:06:29.784008 9951 tablet_bootstrap.cc:492] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: No bootstrap required, opened a new log
I20260810 23:06:29.784088 9951 ts_tablet_manager.cc:1397] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260810 23:06:29.784417 9950 log.cc:824] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:29.785111 9950 tablet_bootstrap.cc:492] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: No bootstrap required, opened a new log
I20260810 23:06:29.785167 9950 ts_tablet_manager.cc:1397] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:06:29.785903 9951 raft_consensus.cc:348] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:29.786079 9951 raft_consensus.cc:374] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:29.786126 9951 raft_consensus.cc:729] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb430ddd29824e619a84a7702896e8cf, State: Initialized, Role: FOLLOWER
I20260810 23:06:29.786270 9951 consensus_queue.cc:260] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:29.786547 9951 ts_tablet_manager.cc:1428] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260810 23:06:29.786583 9793 heartbeater.cc:503] Master 127.23.56.126:34295 was elected leader, sending a full tablet report...
I20260810 23:06:29.787108 9950 raft_consensus.cc:348] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:29.787313 9950 raft_consensus.cc:374] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:06:29.787367 9950 raft_consensus.cc:729] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9262ea11a14e8fb9c19b307914f6dc, State: Initialized, Role: FOLLOWER
I20260810 23:06:29.787508 9950 consensus_queue.cc:260] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:29.787814 9950 ts_tablet_manager.cc:1428] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:06:29.787828 9941 heartbeater.cc:503] Master 127.23.56.126:34295 was elected leader, sending a full tablet report...
W20260810 23:06:29.865103 9794 tablet.cc:2369] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:06:29.899019 9649 raft_consensus.cc:482] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:29.899186 9649 raft_consensus.cc:504] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:29.899595 9649 leader_election.cc:288] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521), eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751)
I20260810 23:06:29.902482 9947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.899884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:49187 (local address 127.23.56.67:36521)
0810 23:06:29.899968 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.899970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.899981 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:06:29.900015 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.900016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.900026 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.900116 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:29.900269 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.900762 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.901724 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.902109 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.902198 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.902206 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.902369 (+ 163us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.902409 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 23:06:29.902482 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.899768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:36521 (local address 127.23.56.65:49187)
0810 23:06:29.899844 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:29.899857 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:29.899916 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:29.900127 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:29.900128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:29.900130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:29.900193 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.900195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.900850 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.900851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.901626 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.901628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.902195 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.902196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.902274 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.902293 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 23:06:29.902363 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:06:29.902406 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 23:06:29.902664 9948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.900000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:53007 (local address 127.23.56.66:33751)
0810 23:06:29.900055 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.900057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.900186 (+ 129us) server_negotiation.cc:407] Connection header received
0810 23:06:29.900251 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.900252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.900266 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.900340 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:06:29.900510 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.900966 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.901917 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.902298 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.902382 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.902390 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.902556 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.902595 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 23:06:29.902680 9956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.899909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:33751 (local address 127.23.56.65:53007)
0810 23:06:29.900113 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:29.900121 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:29.900194 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:29.900343 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:29.900344 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:29.900346 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:06:29.900430 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.900432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.901058 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.901059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.901818 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:29.901820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.902401 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.902402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:29.902473 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.902476 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:06:29.902554 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 23:06:29.902592 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":131,"threads_started":1}
I20260810 23:06:29.902881 9892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e9181927ae493b8cfa9e66808e811b" candidate_uuid: "92894752a062462482fcde96416d4d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" is_pre_election: true
I20260810 23:06:29.903045 9892 raft_consensus.cc:2454] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92894752a062462482fcde96416d4d38 in term 0.
I20260810 23:06:29.903198 9746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e9181927ae493b8cfa9e66808e811b" candidate_uuid: "92894752a062462482fcde96416d4d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb430ddd29824e619a84a7702896e8cf" is_pre_election: true
I20260810 23:06:29.903265 9520 leader_election.cc:302] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9262ea11a14e8fb9c19b307914f6dc, 92894752a062462482fcde96416d4d38; no voters:
I20260810 23:06:29.903358 9746 raft_consensus.cc:2454] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92894752a062462482fcde96416d4d38 in term 0.
I20260810 23:06:29.903394 9649 raft_consensus.cc:2781] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:06:29.903479 9649 raft_consensus.cc:482] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:06:29.903509 9649 raft_consensus.cc:3037] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:29.904232 9649 raft_consensus.cc:504] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:29.904361 9649 leader_election.cc:288] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 1 election: Requested vote from peers 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521), eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751)
I20260810 23:06:29.904548 9892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e9181927ae493b8cfa9e66808e811b" candidate_uuid: "92894752a062462482fcde96416d4d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9262ea11a14e8fb9c19b307914f6dc"
I20260810 23:06:29.904561 9746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81e9181927ae493b8cfa9e66808e811b" candidate_uuid: "92894752a062462482fcde96416d4d38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb430ddd29824e619a84a7702896e8cf"
I20260810 23:06:29.904620 9892 raft_consensus.cc:3037] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:29.904655 9746 raft_consensus.cc:3037] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:06:29.905303 9892 raft_consensus.cc:2454] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92894752a062462482fcde96416d4d38 in term 1.
I20260810 23:06:29.905424 9746 raft_consensus.cc:2454] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92894752a062462482fcde96416d4d38 in term 1.
I20260810 23:06:29.905464 9520 leader_election.cc:302] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9262ea11a14e8fb9c19b307914f6dc, 92894752a062462482fcde96416d4d38; no voters:
I20260810 23:06:29.905539 9649 raft_consensus.cc:2781] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:06:29.905679 9649 raft_consensus.cc:686] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 LEADER]: Becoming Leader. State: Replica: 92894752a062462482fcde96416d4d38, State: Running, Role: LEADER
I20260810 23:06:29.905764 9649 consensus_queue.cc:237] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:29.906628 9435 catalog_manager.cc:5495] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92894752a062462482fcde96416d4d38 (127.23.56.65). New cstate: current_term: 1 leader_uuid: "92894752a062462482fcde96416d4d38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } health_report { overall_health: HEALTHY } } }
I20260810 23:06:29.921744 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.919657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60552 (local address 127.23.56.65:40733)
0810 23:06:29.919726 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:29.919727 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:29.919736 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:06:29.919779 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:29.919779 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:29.919794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:29.919845 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:06:29.919985 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.920436 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.921100 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:29.921272 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:29.921319 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:29.921435 (+ 116us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:06:29.921523 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:06:29.921574 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:29.921643 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 23:06:29.921667 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:06:29.923034 9892 raft_consensus.cc:1261] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [term 1 FOLLOWER]: Refusing update from remote peer 92894752a062462482fcde96416d4d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:29.923111 9746 raft_consensus.cc:1261] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [term 1 FOLLOWER]: Refusing update from remote peer 92894752a062462482fcde96416d4d38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:06:29.923391 9649 consensus_queue.cc:1035] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:29.923548 9957 consensus_queue.cc:1035] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:06:29.926252 9961 mvcc.cc:205] Tried to move back new op lower bound from 7317107465922613248 to 7317107465854705664. Current Snapshot: MvccSnapshot[applied={T|T < 7317107465922613248}]
I20260810 23:06:29.926482 9962 mvcc.cc:205] Tried to move back new op lower bound from 7317107465922613248 to 7317107465854705664. Current Snapshot: MvccSnapshot[applied={T|T < 7317107465922613248}]
I20260810 23:06:29.926771 9963 mvcc.cc:205] Tried to move back new op lower bound from 7317107465922613248 to 7317107465854705664. Current Snapshot: MvccSnapshot[applied={T|T < 7317107465922613248}]
I20260810 23:06:29.928124 9434 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:44106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:29.928203 9434 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:44106:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:06:29.928902 9434 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Sending DeleteTablet for 3 replicas of tablet 81e9181927ae493b8cfa9e66808e811b
I20260810 23:06:29.929263 9572 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e9181927ae493b8cfa9e66808e811b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:29 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:60538
I20260810 23:06:29.929275 9725 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e9181927ae493b8cfa9e66808e811b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:29 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:55600
I20260810 23:06:29.929299 9871 tablet_service.cc:1514] Processing DeleteTablet for tablet 81e9181927ae493b8cfa9e66808e811b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:06:29 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:58070
I20260810 23:06:29.929352 9652 tablet_replica.cc:330] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: stopping tablet replica
I20260810 23:06:29.929405 9652 raft_consensus.cc:2229] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:06:29.929490 9652 raft_consensus.cc:2258] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:29.929541 9972 tablet_replica.cc:330] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: stopping tablet replica
I20260810 23:06:29.929641 9972 raft_consensus.cc:2229] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:29.929734 9972 raft_consensus.cc:2258] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:29.929787 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 9480
I20260810 23:06:29.933434 9972 ts_tablet_manager.cc:1905] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:29.934513 9970 tablet_replica.cc:330] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: stopping tablet replica
I20260810 23:06:29.934635 9970 raft_consensus.cc:2229] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:06:29.934749 9970 raft_consensus.cc:2258] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:06:29.935351 9972 ts_tablet_manager.cc:1918] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:29.935417 9972 log.cc:1196] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/81e9181927ae493b8cfa9e66808e811b
I20260810 23:06:29.935704 9972 ts_tablet_manager.cc:1939] T 81e9181927ae493b8cfa9e66808e811b P 5e9262ea11a14e8fb9c19b307914f6dc: Deleting consensus metadata
I20260810 23:06:29.935734 9970 ts_tablet_manager.cc:1905] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:06:29.936304 9419 catalog_manager.cc:4841] TS 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521): tablet 81e9181927ae493b8cfa9e66808e811b (table test-table [id=96319cead07441209954f5c2e66e1c72]) successfully deleted
I20260810 23:06:29.937404 9970 ts_tablet_manager.cc:1918] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:06:29.937466 9970 log.cc:1196] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/81e9181927ae493b8cfa9e66808e811b
I20260810 23:06:29.937768 9970 ts_tablet_manager.cc:1939] T 81e9181927ae493b8cfa9e66808e811b P eb430ddd29824e619a84a7702896e8cf: Deleting consensus metadata
I20260810 23:06:29.938158 9419 catalog_manager.cc:4841] TS eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751): tablet 81e9181927ae493b8cfa9e66808e811b (table test-table [id=96319cead07441209954f5c2e66e1c72]) successfully deleted
W20260810 23:06:29.944344 9419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:40733 (error 108)
I20260810 23:06:29.944849 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 9648
I20260810 23:06:29.944896 9946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:29.944774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:40733 (local address 127.0.0.1:60556)
0810 23:06:29.944829 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:29.944866 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:40733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 23:06:29.946979 9419 catalog_manager.cc:4584] TS 92894752a062462482fcde96416d4d38 (127.23.56.65:40733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81e9181927ae493b8cfa9e66808e811b: Network error: Client connection negotiation failed: client connection to 127.23.56.65:40733: connect: Connection refused (error 111)
I20260810 23:06:29.952490 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 9796
I20260810 23:06:29.958031 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 9400
I20260810 23:06:49.964564 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34295
--webserver_interface=127.23.56.126
--webserver_port=45647
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:34295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:50.047649 9974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:50.047895 9974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:50.047935 9974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:50.049068 9974 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 23:06:50.049134 9974 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 23:06:50.049155 9974 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 23:06:50.049173 9974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:06:50.049207 9974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:50.049232 9974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:50.049264 9974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:06:50.049293 9974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:06:50.050748 9974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:34295
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:34295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=45647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:50.051028 9974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:50.051287 9974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:06:50.056631 9974 server_base.cc:1034] running on GCE node
W20260810 23:06:50.056748 9980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:50.056757 9981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:50.056883 9983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:50.057152 9974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:50.057446 9974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:50.058594 9974 hybrid_clock.cc:648] HybridClock initialized: now 1786403210058566 us; error 40 us; skew 500 ppm
Aug 10 23:06:50 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:50.061157 9974 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:06:50.061625 9974 webserver.cc:460] Webserver started at http://127.23.56.126:45647/ using document root <none> and password file <none>
I20260810 23:06:50.061848 9974 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:50.061896 9974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:50.063261 9974 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:50.063956 9991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:50.064126 9974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:06:50.064224 9974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "fd57bf2d6ed04d35a162af020f6f5f86"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "c4a5e16954dd72fc2fddb34e9852c0ce"
server_key_iv: "21918d8104e4b10c3b1ae7b0c6fca2a9"
server_key_version: "encryptionkey@0"
I20260810 23:06:50.064582 9974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:50.080090 9974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:50.081881 9974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:50.082047 9974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:50.086273 9974 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:34295
I20260810 23:06:50.086318 10047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:34295 every 8 connection(s)
I20260810 23:06:50.086704 9974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:06:50.088277 10048 sys_catalog.cc:263] Verifying existing consensus state
I20260810 23:06:50.088922 10048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Bootstrap starting.
I20260810 23:06:50.090368 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 9974
I20260810 23:06:50.090633 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:40733
--local_ip_for_outbound_sockets=127.23.56.65
--tserver_master_addrs=127.23.56.126:34295
--webserver_port=36015
--webserver_interface=127.23.56.65
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:50.098186 10048 log.cc:824] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:50.100587 10048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:06:50.100922 10048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Bootstrap complete.
I20260810 23:06:50.103413 10048 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:50.103641 10048 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd57bf2d6ed04d35a162af020f6f5f86, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.103839 10048 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:50.103906 10048 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:06:50.103958 10048 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:06:50.104009 10048 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 2 FOLLOWER]: Advancing to term 3
I20260810 23:06:50.105105 10048 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:50.105225 10048 leader_election.cc:302] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd57bf2d6ed04d35a162af020f6f5f86; no voters:
I20260810 23:06:50.105444 10048 leader_election.cc:288] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 23:06:50.105481 10052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 23:06:50.105611 10052 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [term 3 LEADER]: Becoming Leader. State: Replica: fd57bf2d6ed04d35a162af020f6f5f86, State: Running, Role: LEADER
I20260810 23:06:50.105706 10052 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } }
I20260810 23:06:50.105739 10048 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:06:50.106025 10053 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } } }
I20260810 23:06:50.106240 10053 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:50.106026 10052 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd57bf2d6ed04d35a162af020f6f5f86. Latest consensus state: current_term: 3 leader_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd57bf2d6ed04d35a162af020f6f5f86" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 34295 } } }
I20260810 23:06:50.106555 10052 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86 [sys.catalog]: This master's current role is: LEADER
I20260810 23:06:50.106691 10059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:06:50.107667 10059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:06:50.107852 10059 catalog_manager.cc:1259] Loaded cluster ID: 6dfac5861d724782a0eeb8117fb88f5d
I20260810 23:06:50.108042 10059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:06:50.108927 10059 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:06:50.109125 10059 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Loaded TSK: 0
I20260810 23:06:50.109187 10059 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fd57bf2d6ed04d35a162af020f6f5f86: Loaded TSK: 1
F20260810 23:06:50.109423 10059 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe5cf15e9ad google::LogMessage::Fail() at ??:0
@ 0x7fe5cf16188a google::LogMessage::SendToLog() at ??:0
@ 0x7fe5cf15e4de google::LogMessage::Flush() at ??:0
@ 0x7fe5cf15fa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe5d18cb104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe5d18c8d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe5d70b81cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe5d70b848e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe5d70b8594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fe5d707ec48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fe5d7088ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe5d102c0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe5d1021954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe5d1e856db start_thread at ??:0
@ 0x7fe5cdc5c61f clone at ??:0
W20260810 23:06:50.210108 10050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:50.210402 10050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:50.210443 10050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:50.212182 10050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:50.212249 10050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:50.212291 10050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:06:50.214501 10050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:40733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=36015
--tserver_master_addrs=127.23.56.126:34295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:50.214772 10050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:50.215054 10050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:50.221120 10091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:50.221158 10090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:50.221107 10093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:50.221283 10050 server_base.cc:1034] running on GCE node
I20260810 23:06:50.221555 10050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:50.221818 10050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:50.223024 10050 hybrid_clock.cc:648] HybridClock initialized: now 1786403210223010 us; error 31 us; skew 500 ppm
Aug 10 23:06:50 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:50.225765 10050 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:06:50.226239 10050 webserver.cc:460] Webserver started at http://127.23.56.65:36015/ using document root <none> and password file <none>
I20260810 23:06:50.226513 10050 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:50.226583 10050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:50.228142 10050 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:50.228912 10101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:50.229089 10050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:50.229197 10050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "92894752a062462482fcde96416d4d38"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "b6b42f08ce689532055409adfa17639c"
server_key_iv: "0f80a81ebd59427f39a3dc92ae7fd226"
server_key_version: "encryptionkey@0"
I20260810 23:06:50.229553 10050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:50.244024 10050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:50.246253 10050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:50.246429 10050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:50.246692 10050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:50.247159 10108 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 23:06:50.249153 10050 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 23:06:50.249215 10050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260810 23:06:50.249259 10050 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 23:06:50.249933 10050 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 23:06:50.249989 10050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260810 23:06:50.250135 10108 tablet_bootstrap.cc:492] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Bootstrap starting.
I20260810 23:06:50.257440 10050 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:40733
I20260810 23:06:50.257587 10223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:40733 every 8 connection(s)
I20260810 23:06:50.257884 10050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:06:50.262405 10108 log.cc:824] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Log is configured to *not* fsync() on all Append() calls
I20260810 23:06:50.265345 10108 tablet_bootstrap.cc:492] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 23:06:50.265722 10108 tablet_bootstrap.cc:492] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Bootstrap complete.
I20260810 23:06:50.267026 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 10050
I20260810 23:06:50.267234 10108 ts_tablet_manager.cc:1397] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.006s
I20260810 23:06:50.267351 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:33751
--local_ip_for_outbound_sockets=127.23.56.66
--tserver_master_addrs=127.23.56.126:34295
--webserver_port=42183
--webserver_interface=127.23.56.66
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:06:50.268837 10108 raft_consensus.cc:348] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:50.269192 10108 raft_consensus.cc:729] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92894752a062462482fcde96416d4d38, State: Initialized, Role: FOLLOWER
I20260810 23:06:50.269387 10108 consensus_queue.cc:260] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:50.269685 10108 ts_tablet_manager.cc:1428] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
W20260810 23:06:50.350353 10226 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:06:50.258685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:57141)
0810 23:06:50.258864 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.258909 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:50.259480 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:50.350194 (+ 90714us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
W20260810 23:06:50.350623 10224 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:34295 (0 consecutive failures): Network error: Failed to ping master at 127.23.56.126:34295: Client connection negotiation failed: client connection to 127.23.56.126:34295: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 23:06:50.350963 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.350843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:54845)
0810 23:06:50.350897 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.350933 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 23:06:50.351334 10226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.351249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:58777)
0810 23:06:50.351296 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.351319 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 23:06:50.351465 10224 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 23:06:50.363246 10228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:50.363435 10228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:50.363458 10228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:50.364568 10228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:50.364601 10228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:50.364627 10228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:06:50.365984 10228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:33751
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=42183
--tserver_master_addrs=127.23.56.126:34295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:50.366159 10228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:50.366338 10228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:50.371297 10234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:50.371299 10235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:50.371308 10238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:50.371322 10228 server_base.cc:1034] running on GCE node
I20260810 23:06:50.371703 10228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:50.371969 10228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:50.373109 10228 hybrid_clock.cc:648] HybridClock initialized: now 1786403210373098 us; error 27 us; skew 500 ppm
Aug 10 23:06:50 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:50.375747 10228 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:06:50.376174 10228 webserver.cc:460] Webserver started at http://127.23.56.66:42183/ using document root <none> and password file <none>
I20260810 23:06:50.376400 10228 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:50.376468 10228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:50.377729 10228 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:06:50.378302 10246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:50.378470 10228 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:50.378561 10228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "eb430ddd29824e619a84a7702896e8cf"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "40da989c97720270ae705056e2d2ebba"
server_key_iv: "5d7d72d6eeeebc9427959a78389bbfa9"
server_key_version: "encryptionkey@0"
I20260810 23:06:50.378911 10228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:50.397549 10228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:50.399402 10228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:50.399564 10228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:50.399796 10228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:50.400105 10228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:50.400161 10228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:50.400200 10228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:50.400228 10228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:50.406236 10228 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:33751
I20260810 23:06:50.406286 10369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:33751 every 8 connection(s)
I20260810 23:06:50.406602 10228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:06:50.407626 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.407266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:60961)
0810 23:06:50.407419 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.407489 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
W20260810 23:06:50.408070 10370 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:34295 (0 consecutive failures): Network error: Failed to ping master at 127.23.56.126:34295: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
I20260810 23:06:50.408356 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.408254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:57349)
0810 23:06:50.408303 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.408328 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 23:06:50.408630 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.408547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:41891)
0810 23:06:50.408590 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.408612 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 23:06:50.408739 10370 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:06:50.413908 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 10228
I20260810 23:06:50.414199 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:36521
--local_ip_for_outbound_sockets=127.23.56.67
--tserver_master_addrs=127.23.56.126:34295
--webserver_port=34611
--webserver_interface=127.23.56.67
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:39169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:06:50.495371 10373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:06:50.495623 10373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:06:50.495661 10373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:06:50.497500 10373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:06:50.497567 10373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:06:50.497606 10373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:06:50.499197 10373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:39169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:36521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=34611
--tserver_master_addrs=127.23.56.126:34295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:06:50.499444 10373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:06:50.499691 10373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:06:50.504868 10381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:50.504920 10373 server_base.cc:1034] running on GCE node
W20260810 23:06:50.504820 10378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:06:50.504820 10379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:06:50.505304 10373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:06:50.505514 10373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:06:50.506652 10373 hybrid_clock.cc:648] HybridClock initialized: now 1786403210506642 us; error 27 us; skew 500 ppm
Aug 10 23:06:50 dist-test-slave-1xrh krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:06:50.509346 10373 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:06:50.509802 10373 webserver.cc:460] Webserver started at http://127.23.56.67:34611/ using document root <none> and password file <none>
I20260810 23:06:50.510053 10373 fs_manager.cc:359] Metadata directory not provided
I20260810 23:06:50.510128 10373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:06:50.511523 10373 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:50.512177 10389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:50.512378 10373 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:06:50.512485 10373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "5e9262ea11a14e8fb9c19b307914f6dc"
format_stamp: "Formatted at 2026-08-10 23:06:08 on dist-test-slave-1xrh"
server_key: "0b8074a1e2525fed08b48e2838982240"
server_key_iv: "eced417ae5d833041c12532a7344c593"
server_key_version: "encryptionkey@0"
I20260810 23:06:50.512814 10373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:06:50.535917 10373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:06:50.538266 10373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:06:50.538436 10373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:06:50.538671 10373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:06:50.538992 10373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:06:50.539047 10373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:50.539090 10373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:06:50.539137 10373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:06:50.544942 10373 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:36521
I20260810 23:06:50.544998 10511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:36521 every 8 connection(s)
I20260810 23:06:50.545399 10373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:06:50.546430 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.546070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:55261)
0810 23:06:50.546222 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.546290 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
W20260810 23:06:50.546840 10512 heartbeater.cc:644] Failed to heartbeat to 127.23.56.126:34295 (0 consecutive failures): Network error: Failed to ping master at 127.23.56.126:34295: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
I20260810 23:06:50.547114 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.547023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:36243)
0810 23:06:50.547069 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.547096 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 23:06:50.547464 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:50.547383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:58387)
0810 23:06:50.547428 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:50.547450 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 23:06:50.547585 10512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 23:06:50.549944 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 10373
I20260810 23:06:51.352146 10516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:51.351883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:41621)
0810 23:06:51.352040 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:51.352081 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":74,"threads_started":1}
I20260810 23:06:51.409235 10517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:51.409028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:52817)
0810 23:06:51.409136 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:51.409167 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260810 23:06:51.548069 10518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:51.547884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:40625)
0810 23:06:51.547988 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:51.548018 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
I20260810 23:06:51.991966 10519 raft_consensus.cc:482] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:51.992100 10519 raft_consensus.cc:504] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:51.992439 10519 leader_election.cc:288] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521), eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751)
Aug 10 23:06:51 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.66@KRBTEST.COM
Aug 10 23:06:51 dist-test-slave-1xrh krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403210, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.67@KRBTEST.COM
I20260810 23:06:52.003846 10523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:51.992985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:45283 (local address 127.23.56.66:33751)
0810 23:06:51.993133 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:51.993150 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:51.993169 (+ 19us) server_negotiation.cc:407] Connection header received
0810 23:06:51.993636 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:51.993639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:51.993678 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:51.993845 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:51.994569 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:51.995498 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:51.996684 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:51.997076 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:51.997186 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:51.999542 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:51.999549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:51.999549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:51.999565 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:52.001628 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:52.002282 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:52.002285 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:52.002285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:52.002402 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:52.002756 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:52.002757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:52.002758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:52.003095 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:52.003234 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:52.003569 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:06:52.003710 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260810 23:06:52.003896 10521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:51.992783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:44083 (local address 127.23.56.67:36521)
0810 23:06:51.992999 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 23:06:51.993013 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:06:51.993050 (+ 37us) server_negotiation.cc:407] Connection header received
0810 23:06:51.993570 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:06:51.993573 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:06:51.993621 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:06:51.993743 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:06:51.994584 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:51.995491 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:51.996655 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:06:51.997080 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:06:51.997220 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:51.999920 (+ 2700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:06:51.999928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:06:51.999928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:06:51.999939 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:06:52.001791 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:52.002281 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:52.002285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:52.002285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:52.002397 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:06:52.002756 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:06:52.002757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:06:52.002758 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:06:52.003095 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:06:52.003234 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 23:06:52.003569 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 23:06:52.003710 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":88,"threads_started":1}
I20260810 23:06:52.003886 10520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:51.992635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:36521 (local address 127.23.56.65:44083)
0810 23:06:51.992770 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:51.992784 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:51.992860 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:51.993823 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:51.993827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:51.994232 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:51.994435 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:51.994441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:51.995597 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:51.995599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:51.996571 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:51.996574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:51.997190 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:51.997191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:51.997282 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:51.997933 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:51.997937 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:51.999803 (+ 1866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:52.001865 (+ 2062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:52.001866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:52.001867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:52.002189 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:52.002477 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:52.002478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:52.002478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:52.002650 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:52.003194 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:52.003202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:52.003365 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 23:06:52.003489 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 23:06:52.003708 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":18,"thread_start_us":67,"threads_started":1}
I20260810 23:06:52.004308 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:51.992833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:33751 (local address 127.23.56.65:45283)
0810 23:06:51.992945 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:51.992958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:06:51.993017 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:06:51.993875 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:06:51.993876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:06:51.994231 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:06:51.994435 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:51.994441 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:51.995613 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:51.995614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:51.996571 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:06:51.996574 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:06:51.997168 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:06:51.997170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:06:51.997282 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:06:51.997870 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:06:51.997901 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:06:51.999420 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:06:52.001728 (+ 2308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:52.001735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:52.001737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:52.002189 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:52.002477 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:06:52.002478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:06:52.002478 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:06:52.002650 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:06:52.003194 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:06:52.003202 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:06:52.003365 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 23:06:52.003489 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 23:06:52.003765 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"mutex_wait_us":225,"thread_start_us":69,"threads_started":1}
W20260810 23:06:52.004964 10103 leader_election.cc:341] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521): Not found: Tablet not found: 81e9181927ae493b8cfa9e66808e811b
W20260810 23:06:52.005085 10103 leader_election.cc:341] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751): Not found: Tablet not found: 81e9181927ae493b8cfa9e66808e811b
I20260810 23:06:52.005122 10103 leader_election.cc:302] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92894752a062462482fcde96416d4d38; no voters: 5e9262ea11a14e8fb9c19b307914f6dc, eb430ddd29824e619a84a7702896e8cf
I20260810 23:06:52.005256 10519 raft_consensus.cc:2726] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:06:52.352737 10522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:52.352588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:51519)
0810 23:06:52.352668 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:52.352697 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 23:06:52.409848 10524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:52.409647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:53823)
0810 23:06:52.409764 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:52.409807 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260810 23:06:52.548645 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:52.548468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:58265)
0810 23:06:52.548576 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:52.548608 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":56,"threads_started":1}
I20260810 23:06:53.353422 10526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:53.353203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:41367)
0810 23:06:53.353337 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:53.353376 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":61,"threads_started":1}
I20260810 23:06:53.410451 10528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:53.410264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:54111)
0810 23:06:53.410377 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:53.410412 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260810 23:06:53.549300 10529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:53.549065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:40341)
0810 23:06:53.549225 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:53.549263 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":80,"threads_started":1}
I20260810 23:06:54.136889 10530 raft_consensus.cc:482] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:54.136989 10530 raft_consensus.cc:504] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:54.137202 10530 leader_election.cc:288] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521), eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751)
W20260810 23:06:54.137737 10103 leader_election.cc:341] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751): Not found: Tablet not found: 81e9181927ae493b8cfa9e66808e811b
W20260810 23:06:54.137823 10103 leader_election.cc:341] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521): Not found: Tablet not found: 81e9181927ae493b8cfa9e66808e811b
I20260810 23:06:54.137843 10103 leader_election.cc:302] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92894752a062462482fcde96416d4d38; no voters: 5e9262ea11a14e8fb9c19b307914f6dc, eb430ddd29824e619a84a7702896e8cf
I20260810 23:06:54.137986 10530 raft_consensus.cc:2726] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:06:54.354079 10532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.353870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:41089)
0810 23:06:54.354002 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:54.354031 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260810 23:06:54.411195 10533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.410930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:45687)
0810 23:06:54.411054 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:54.411087 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260810 23:06:54.550067 10534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:54.549817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:51267)
0810 23:06:54.549981 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:54.550012 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260810 23:06:55.354844 10536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.354586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:52163)
0810 23:06:55.354722 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.354761 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260810 23:06:55.411808 10537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.411604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:32875)
0810 23:06:55.411735 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.411767 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":75,"threads_started":1}
I20260810 23:06:55.550683 10538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:55.550475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:56001)
0810 23:06:55.550609 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:55.550643 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260810 23:06:55.851267 10539 raft_consensus.cc:482] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:55.851366 10539 raft_consensus.cc:504] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:55.851529 10539 leader_election.cc:288] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521), eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751)
W20260810 23:06:55.851999 10103 leader_election.cc:341] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521): Not found: Tablet not found: 81e9181927ae493b8cfa9e66808e811b
W20260810 23:06:55.852092 10103 leader_election.cc:341] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751): Not found: Tablet not found: 81e9181927ae493b8cfa9e66808e811b
I20260810 23:06:55.852109 10103 leader_election.cc:302] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92894752a062462482fcde96416d4d38; no voters: 5e9262ea11a14e8fb9c19b307914f6dc, eb430ddd29824e619a84a7702896e8cf
I20260810 23:06:55.852180 10539 raft_consensus.cc:2726] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:06:56.355671 10540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:56.355469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:60147)
0810 23:06:56.355587 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:56.355622 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
I20260810 23:06:56.412407 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:56.412199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:33921)
0810 23:06:56.412329 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:56.412365 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":62,"threads_started":1}
I20260810 23:06:56.551362 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:56.551166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:56281)
0810 23:06:56.551289 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:56.551320 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
I20260810 23:06:57.356406 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:57.356171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:43909)
0810 23:06:57.356314 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:57.356355 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260810 23:06:57.413149 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:57.412850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:45393)
0810 23:06:57.413074 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:57.413107 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":91,"threads_started":1}
I20260810 23:06:57.552043 10545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:57.551795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:39437)
0810 23:06:57.551942 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:57.551981 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":50,"threads_started":1}
I20260810 23:06:58.357160 10546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:58.356929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:37521)
0810 23:06:58.357074 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:58.357113 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":63,"threads_started":1}
I20260810 23:06:58.413877 10547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:58.413658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:57619)
0810 23:06:58.413792 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:58.413830 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":62,"threads_started":1}
I20260810 23:06:58.552762 10548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:58.552562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:45091)
0810 23:06:58.552684 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:58.552718 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260810 23:06:59.357826 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:59.357599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:34333)
0810 23:06:59.357740 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:59.357777 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":62,"threads_started":1}
I20260810 23:06:59.414534 10550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:59.414308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:37205)
0810 23:06:59.414462 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:59.414495 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260810 23:06:59.553455 10551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:06:59.553235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:39025)
0810 23:06:59.553377 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 23:06:59.553413 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":58,"threads_started":1}
I20260810 23:06:59.794117 10552 raft_consensus.cc:482] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:06:59.794222 10552 raft_consensus.cc:504] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e9262ea11a14e8fb9c19b307914f6dc" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 36521 } } peers { permanent_uuid: "eb430ddd29824e619a84a7702896e8cf" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 33751 } } peers { permanent_uuid: "92894752a062462482fcde96416d4d38" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 40733 } }
I20260810 23:06:59.794430 10552 leader_election.cc:288] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521), eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751)
W20260810 23:06:59.794886 10103 leader_election.cc:341] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5e9262ea11a14e8fb9c19b307914f6dc (127.23.56.67:36521): Not found: Tablet not found: 81e9181927ae493b8cfa9e66808e811b
W20260810 23:06:59.794986 10103 leader_election.cc:341] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eb430ddd29824e619a84a7702896e8cf (127.23.56.66:33751): Not found: Tablet not found: 81e9181927ae493b8cfa9e66808e811b
I20260810 23:06:59.795035 10103 leader_election.cc:302] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 92894752a062462482fcde96416d4d38; no voters: 5e9262ea11a14e8fb9c19b307914f6dc, eb430ddd29824e619a84a7702896e8cf
I20260810 23:06:59.795158 10552 raft_consensus.cc:2726] T 81e9181927ae493b8cfa9e66808e811b P 92894752a062462482fcde96416d4d38 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 23:07:00.358551 10553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:00.358320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:40921)
0810 23:07:00.358466 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:00.358504 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260810 23:07:00.415259 10554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:00.414972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:38101)
0810 23:07:00.415170 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:00.415212 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260810 23:07:00.554116 10555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:00.553914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:48115)
0810 23:07:00.554039 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:00.554075 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260810 23:07:01.359330 10556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:01.359023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:54505)
0810 23:07:01.359241 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:01.359281 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":83,"threads_started":1}
I20260810 23:07:01.415927 10557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:01.415701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:50933)
0810 23:07:01.415840 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:01.415883 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":61,"threads_started":1}
I20260810 23:07:01.554773 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:01.554552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:52141)
0810 23:07:01.554678 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:01.554722 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260810 23:07:02.360024 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:02.359803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:54941)
0810 23:07:02.359934 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:02.359974 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260810 23:07:02.416599 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:02.416379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:58571)
0810 23:07:02.416520 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:02.416555 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":78,"threads_started":1}
I20260810 23:07:02.555516 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:02.555282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:34375)
0810 23:07:02.555432 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:02.555468 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260810 23:07:03.360678 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:03.360472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:33777)
0810 23:07:03.360592 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:03.360628 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
I20260810 23:07:03.417272 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:03.417078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:55491)
0810 23:07:03.417196 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:03.417228 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260810 23:07:03.556152 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:03.555952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:43751)
0810 23:07:03.556071 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:03.556106 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260810 23:07:04.361454 10565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:04.361223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:41345)
0810 23:07:04.361357 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:04.361405 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":68,"threads_started":1}
I20260810 23:07:04.417917 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:04.417711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:35475)
0810 23:07:04.417835 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:04.417870 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260810 23:07:04.556785 10567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:04.556579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:46749)
0810 23:07:04.556702 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:04.556739 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260810 23:07:05.362149 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:05.361947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.65:37397)
0810 23:07:05.362064 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:05.362098 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260810 23:07:05.418620 10569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:05.418392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.66:35563)
0810 23:07:05.418545 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:05.418579 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260810 23:07:05.557497 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:05.557272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:34295 (local address 127.23.56.67:52843)
0810 23:07:05.557415 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:05.557452 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.126:34295: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":78,"threads_started":1}
I20260810 23:07:05.559206 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 10050
I20260810 23:07:05.566458 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 10228
I20260810 23:07:05.572304 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 10373
2026-08-10T23:07:05Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58433 ms)
[----------] 30 tests from SecurityITest (232919 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:05 dist-test-slave-1xrh krb5kdc[10573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:05 dist-test-slave-1xrh krb5kdc[10573](info): set up 2 sockets
Aug 10 23:07:05 dist-test-slave-1xrh krb5kdc[10573](info): commencing operation
krb5kdc: starting...
W20260810 23:07:06.688086 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.094s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:06 dist-test-slave-1xrh krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:06.813871 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33523
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:06.895620 10586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:06.895805 10586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:06.895826 10586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:06.896879 10586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:06.896917 10586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:06.896932 10586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:06.896945 10586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:06.896957 10586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:06.898316 10586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:06.898502 10586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:06.898665 10586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:06.903714 10593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:06.903728 10595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:06.903764 10586 server_base.cc:1034] running on GCE node
W20260810 23:07:06.903726 10592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:06.904199 10586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:06.904435 10586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:06.905575 10586 hybrid_clock.cc:648] HybridClock initialized: now 1786403226905540 us; error 51 us; skew 500 ppm
Aug 10 23:07:06 dist-test-slave-1xrh krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403226, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:06.908104 10586 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:06.908528 10586 webserver.cc:460] Webserver started at http://127.23.56.126:35923/ using document root <none> and password file <none>
I20260810 23:07:06.908764 10586 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:06.908831 10586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:06.908983 10586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:06.909972 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "2442987a38194af7b450656a6d946709"
format_stamp: "Formatted at 2026-08-10 23:07:06 on dist-test-slave-1xrh"
server_key: "20dd3b86565cb89a41785c88b5dc493c"
server_key_iv: "e3c576018090493c2305aa4fb85304a1"
server_key_version: "encryptionkey@0"
I20260810 23:07:06.910316 10586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "2442987a38194af7b450656a6d946709"
format_stamp: "Formatted at 2026-08-10 23:07:06 on dist-test-slave-1xrh"
server_key: "20dd3b86565cb89a41785c88b5dc493c"
server_key_iv: "e3c576018090493c2305aa4fb85304a1"
server_key_version: "encryptionkey@0"
I20260810 23:07:06.911682 10586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:07:06.912393 10602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:06.912562 10586 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:06.912667 10586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "2442987a38194af7b450656a6d946709"
format_stamp: "Formatted at 2026-08-10 23:07:06 on dist-test-slave-1xrh"
server_key: "20dd3b86565cb89a41785c88b5dc493c"
server_key_iv: "e3c576018090493c2305aa4fb85304a1"
server_key_version: "encryptionkey@0"
I20260810 23:07:06.912739 10586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:06.921228 10586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:06.923237 10586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:06.923394 10586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:06.927111 10586 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33523
I20260810 23:07:06.927196 10661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33523 every 8 connection(s)
I20260810 23:07:06.927577 10586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:06.928419 10662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:06.928614 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 10586
I20260810 23:07:06.928715 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:06.928937 23777 external_mini_cluster.cc:1428] Setting key 0af711ac7c7692b06b5276a29ff66316
I20260810 23:07:06.932322 10662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709: Bootstrap starting.
I20260810 23:07:06.932963 10662 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:06.933267 10662 log.cc:824] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:06.934016 10662 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709: No bootstrap required, opened a new log
Aug 10 23:07:06 dist-test-slave-1xrh krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403226, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:06.936302 10662 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2442987a38194af7b450656a6d946709" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33523 } }
I20260810 23:07:06.936492 10662 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:06.936539 10662 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2442987a38194af7b450656a6d946709, State: Initialized, Role: FOLLOWER
I20260810 23:07:06.936677 10662 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2442987a38194af7b450656a6d946709" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33523 } }
I20260810 23:07:06.936764 10662 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:06.936786 10662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:06.936861 10662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:06.937534 10662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2442987a38194af7b450656a6d946709" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33523 } }
I20260810 23:07:06.937624 10662 leader_election.cc:302] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2442987a38194af7b450656a6d946709; no voters:
I20260810 23:07:06.937868 10662 leader_election.cc:288] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:06.937898 10667 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:06.938014 10667 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [term 1 LEADER]: Becoming Leader. State: Replica: 2442987a38194af7b450656a6d946709, State: Running, Role: LEADER
I20260810 23:07:06.938140 10662 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:06.938153 10667 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2442987a38194af7b450656a6d946709" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33523 } }
I20260810 23:07:06.938637 10667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2442987a38194af7b450656a6d946709. Latest consensus state: current_term: 1 leader_uuid: "2442987a38194af7b450656a6d946709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2442987a38194af7b450656a6d946709" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33523 } } }
I20260810 23:07:06.938797 10667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:06.938840 10669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2442987a38194af7b450656a6d946709" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2442987a38194af7b450656a6d946709" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33523 } } }
I20260810 23:07:06.938938 10669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709 [sys.catalog]: This master's current role is: LEADER
W20260810 23:07:06.940013 10693 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:07:06.940069 10693 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:07:06.940122 10679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:06.940268 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:06.929442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.23.56.126:33523)
0810 23:07:06.929624 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:06.929631 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:06.929655 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:07:06.930290 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:06.930299 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:06.930358 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:06.930484 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:06.931494 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.932416 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.933219 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:06.933669 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:06.933807 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:06.936145 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:06.936149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:06.936149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:06.936160 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:06.938049 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:06.938496 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:06.938498 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:06.938498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:06.938596 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:06.939008 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:06.939009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:06.939009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:06.939348 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:06.939608 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:06.939749 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 23:07:06.940020 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260810 23:07:06.940296 10679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:06.942117 10679 catalog_manager.cc:1347] Generated new cluster ID: 318981b7fc5d42afabe035ef981d7795
I20260810 23:07:06.942168 10679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:06.990123 10679 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:06.990665 10679 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:06.994513 10679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709: Generated new TSK 0
I20260810 23:07:06.994699 10679 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:07:07.029803 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33523
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:07.119943 10698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:07.120158 10698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:07.120184 10698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:07.121536 10698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:07.121584 10698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:07.121613 10698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:07:07.123351 10698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:07.123621 10698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:07.123905 10698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:07:07.129480 10698 server_base.cc:1034] running on GCE node
W20260810 23:07:07.129460 10704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:07.129462 10708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:07.129487 10705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:07.129961 10698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:07.130168 10698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:07.131361 10698 hybrid_clock.cc:648] HybridClock initialized: now 1786403227131341 us; error 32 us; skew 500 ppm
Aug 10 23:07:07 dist-test-slave-1xrh krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:07.133983 10698 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:07:07.134435 10698 webserver.cc:460] Webserver started at http://127.23.56.65:34117/ using document root <none> and password file <none>
I20260810 23:07:07.134631 10698 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:07.134671 10698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:07.134759 10698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:07.135811 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "cafa637d7e9741619209bf926bd7a9e4"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "fc0bb770a5921aaf7f96aca5de251e22"
server_key_iv: "4fa1ef3b0f8d3a84403d71349932ea2b"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.136122 10698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "cafa637d7e9741619209bf926bd7a9e4"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "fc0bb770a5921aaf7f96aca5de251e22"
server_key_iv: "4fa1ef3b0f8d3a84403d71349932ea2b"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.137394 10698 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:07:07.138116 10716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.138298 10698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:07.138384 10698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "cafa637d7e9741619209bf926bd7a9e4"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "fc0bb770a5921aaf7f96aca5de251e22"
server_key_iv: "4fa1ef3b0f8d3a84403d71349932ea2b"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.138455 10698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:07.152181 10698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:07.154500 10698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:07.154614 10698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:07.154790 10698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:07.155110 10698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:07.155205 10698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.155257 10698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:07.155284 10698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.161082 10698 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:38511
I20260810 23:07:07.161163 10842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:38511 every 8 connection(s)
I20260810 23:07:07.161494 10698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:07:07.166039 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 10698
I20260810 23:07:07.166159 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:07:07.166359 23777 external_mini_cluster.cc:1428] Setting key d6219d5a8fb8308555bc868ff40f3408
Aug 10 23:07:07 dist-test-slave-1xrh krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:07.172238 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.162211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33523 (local address 127.23.56.65:45563)
0810 23:07:07.162530 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.162575 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.163101 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.163429 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.163435 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.164117 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:07.164321 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.164325 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.165041 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.165042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.165882 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.165884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.166348 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.166349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.166527 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.167088 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:07.167099 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:07.168835 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:07.170818 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:07.170824 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:07.170826 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:07.171044 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:07.171286 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:07.171287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:07.171287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:07.171398 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:07.171706 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:07.171711 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:07.171876 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 23:07:07.171963 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 23:07:07.172084 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":89,"threads_started":1}
I20260810 23:07:07.172236 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.162302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:45563 (local address 127.23.56.126:33523)
0810 23:07:07.162514 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.162517 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.162677 (+ 160us) server_negotiation.cc:407] Connection header received
0810 23:07:07.163255 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.163257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.163272 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.163341 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:07.164428 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.164941 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.165983 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.166266 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.166334 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.168942 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:07.168943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:07.168944 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:07.168952 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:07.170731 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:07.171148 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:07.171150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:07.171150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:07.171215 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:07.171473 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:07.171474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:07.171475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:07.171636 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:07.171714 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.172015 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.172126 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260810 23:07:07.172857 10843 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33523
I20260810 23:07:07.172948 10843 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:07.173137 10843 heartbeater.cc:511] Master 127.23.56.126:33523 requested a full tablet report, sending...
I20260810 23:07:07.173734 10621 ts_manager.cc:194] Registered new tserver with Master: cafa637d7e9741619209bf926bd7a9e4 (127.23.56.65:38511)
I20260810 23:07:07.174651 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:45563
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:07:07.193686 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33523
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:07.274394 10848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:07.274583 10848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:07.274605 10848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:07.275799 10848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:07.275841 10848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:07.275868 10848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:07:07.277359 10848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:07.277609 10848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:07.277843 10848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:07.283154 10853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:07.283111 10856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:07.283111 10858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:07.283270 10848 server_base.cc:1034] running on GCE node
I20260810 23:07:07.283628 10848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:07.283878 10848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:07.285024 10848 hybrid_clock.cc:648] HybridClock initialized: now 1786403227285015 us; error 31 us; skew 500 ppm
Aug 10 23:07:07 dist-test-slave-1xrh krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:07.287612 10848 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:07:07.288069 10848 webserver.cc:460] Webserver started at http://127.23.56.66:41995/ using document root <none> and password file <none>
I20260810 23:07:07.288326 10848 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:07.288379 10848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:07.288511 10848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:07.289546 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "83d64078687347a9af21bbf08e916f9a"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "ec26896067dc984bf1b74970e41331a1"
server_key_iv: "ea2963322347c2f03f2e624ca66fc7ec"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.289896 10848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "83d64078687347a9af21bbf08e916f9a"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "ec26896067dc984bf1b74970e41331a1"
server_key_iv: "ea2963322347c2f03f2e624ca66fc7ec"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.291287 10848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:07:07.292048 10865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.292254 10848 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:07:07.292346 10848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "83d64078687347a9af21bbf08e916f9a"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "ec26896067dc984bf1b74970e41331a1"
server_key_iv: "ea2963322347c2f03f2e624ca66fc7ec"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.292428 10848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:07.323192 10848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:07.325222 10848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:07.325387 10848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:07.325647 10848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:07.325973 10848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:07.326006 10848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.326061 10848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:07.326090 10848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.332122 10848 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:38303
I20260810 23:07:07.332170 10994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:38303 every 8 connection(s)
I20260810 23:07:07.332480 10848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
Aug 10 23:07:07 dist-test-slave-1xrh krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:07.341535 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 10848
I20260810 23:07:07.341621 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:07:07.341776 23777 external_mini_cluster.cc:1428] Setting key c60ca34a4df6b261db9d635ace391b8b
I20260810 23:07:07.342497 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.333183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:40575 (local address 127.23.56.126:33523)
0810 23:07:07.333246 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.333248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.333454 (+ 206us) server_negotiation.cc:407] Connection header received
0810 23:07:07.334024 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.334025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.334038 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.334088 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:07.335008 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.335563 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.336547 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.336764 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.336849 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.339070 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:07.339071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:07.339072 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:07.339078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:07.340508 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:07.340875 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:07.340876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:07.340876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:07.340929 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:07.341136 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:07.341137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:07.341137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:07.341285 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:07.341354 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.342251 (+ 897us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.342338 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:07:07.342607 10997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.333157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33523 (local address 127.23.56.66:40575)
0810 23:07:07.333335 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.333370 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.333890 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.334152 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.334159 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.334721 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:07.334912 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.334916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.335656 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.335657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.336453 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.336455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.336862 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.336863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.336997 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.337513 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:07.337529 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:07.338988 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:07.340595 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:07.340601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:07.340603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:07.340811 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:07.340989 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:07.340990 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:07.340990 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:07.341082 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:07.341924 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:07.341930 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:07.342089 (+ 159us) client_negotiation.cc:769] Sending connection context
0810 23:07:07.342227 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 23:07:07.342385 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260810 23:07:07.343241 10995 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33523
I20260810 23:07:07.343333 10995 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:07.343526 10995 heartbeater.cc:511] Master 127.23.56.126:33523 requested a full tablet report, sending...
I20260810 23:07:07.343910 10621 ts_manager.cc:194] Registered new tserver with Master: 83d64078687347a9af21bbf08e916f9a (127.23.56.66:38303)
I20260810 23:07:07.344426 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:40575
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:07:07.367245 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33523
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:07.449131 11000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:07.449318 11000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:07.449342 11000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:07.450393 11000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:07.450426 11000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:07.450464 11000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:07:07.451949 11000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33523
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:07.452131 11000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:07.452296 11000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:07.456934 11006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:07.456987 11005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:07.457022 11000 server_base.cc:1034] running on GCE node
W20260810 23:07:07.456991 11008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:07.457319 11000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:07.457505 11000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:07.458652 11000 hybrid_clock.cc:648] HybridClock initialized: now 1786403227458638 us; error 27 us; skew 500 ppm
Aug 10 23:07:07 dist-test-slave-1xrh krb5kdc[10573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:07.461232 11000 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:07:07.461724 11000 webserver.cc:460] Webserver started at http://127.23.56.67:32987/ using document root <none> and password file <none>
I20260810 23:07:07.461980 11000 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:07.462050 11000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:07.462209 11000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:07.463290 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "bf138f2f338b49558e599dedab07afcb"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "faaba64bc359b2ab7a6b960679032f74"
server_key_iv: "2f94f30d6eea75a872c0041bb788bc4e"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.463631 11000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "bf138f2f338b49558e599dedab07afcb"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "faaba64bc359b2ab7a6b960679032f74"
server_key_iv: "2f94f30d6eea75a872c0041bb788bc4e"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.464995 11000 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:07.465745 11017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.465916 11000 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:07.466032 11000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "bf138f2f338b49558e599dedab07afcb"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "faaba64bc359b2ab7a6b960679032f74"
server_key_iv: "2f94f30d6eea75a872c0041bb788bc4e"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.466116 11000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:07.480100 11000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:07.482334 11000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:07.482538 11000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:07.482792 11000 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:07.483103 11000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:07.483178 11000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.483211 11000 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:07.483239 11000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.489078 11000 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:34205
I20260810 23:07:07.489123 11153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:34205 every 8 connection(s)
I20260810 23:07:07.489451 11000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:07:07.492197 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 11000
I20260810 23:07:07.492297 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:07:07.492508 23777 external_mini_cluster.cc:1428] Setting key d0818c61e97398815041bc2c5329055e
Aug 10 23:07:07 dist-test-slave-1xrh krb5kdc[10573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403227, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:07.498345 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.490134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:56393 (local address 127.23.56.126:33523)
0810 23:07:07.490201 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.490203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.490384 (+ 181us) server_negotiation.cc:407] Connection header received
0810 23:07:07.490853 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.490854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.490864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.490926 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:07.491863 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.492427 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.493713 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.493896 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.493967 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.496078 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:07.496080 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:07.496080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:07.496086 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:07.497272 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:07.497621 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:07.497621 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:07.497622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:07.497669 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:07.497862 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:07.497863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:07.497863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:07.497960 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:07.498010 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.498224 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.498284 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:07:07.498500 11156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.490078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33523 (local address 127.23.56.67:56393)
0810 23:07:07.490264 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.490302 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.490773 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.490981 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.490985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.491589 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:07.491784 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.491788 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.492521 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.492523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.493640 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.493643 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.493988 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.493989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.494093 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.494599 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:07.494609 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:07.495996 (+ 1387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:07.497342 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:07.497347 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:07.497349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:07.497557 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:07.497725 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:07.497725 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:07.497725 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:07.497810 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:07.498015 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:07.498020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:07.498119 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 23:07:07.498196 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 23:07:07.498305 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":119,"threads_started":1}
I20260810 23:07:07.498960 11154 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33523
I20260810 23:07:07.499037 11154 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:07.499217 11154 heartbeater.cc:511] Master 127.23.56.126:33523 requested a full tablet report, sending...
I20260810 23:07:07.499540 10621 ts_manager.cc:194] Registered new tserver with Master: bf138f2f338b49558e599dedab07afcb (127.23.56.67:34205)
I20260810 23:07:07.499938 10621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:56393
I20260810 23:07:07.503894 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:07:07.510562 10665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.505195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60132 (local address 127.23.56.126:33523)
0810 23:07:07.505246 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.505247 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.505346 (+ 99us) server_negotiation.cc:407] Connection header received
0810 23:07:07.505396 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.505397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.505416 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.505464 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:07.505956 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.506391 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.507074 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.507298 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.507349 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.508373 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:07.508374 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:07.508374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:07.508380 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:07.509631 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:07.509939 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:07.509940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:07.509940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:07.509989 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:07.510153 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:07.510153 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:07.510153 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:07.510285 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:07.510356 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.510442 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.510502 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260810 23:07:07.512008 10621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:07:07.513597 10621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:07:07.520251 11168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.515973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:34205 (local address 127.0.0.1:47864)
0810 23:07:07.516382 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.516392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.516432 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.516742 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.516743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.516744 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:07.516780 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.516782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.518113 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.518113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.519266 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.519268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.519934 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.519935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.520026 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.520041 (+ 15us) client_negotiation.cc:769] Sending connection context
0810 23:07:07.520110 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 23:07:07.520161 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":29,"thread_start_us":79,"threads_started":1}
I20260810 23:07:07.520363 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.516259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41978 (local address 127.23.56.65:38511)
0810 23:07:07.516453 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.516458 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.516468 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:07.516512 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.516514 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.516526 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.516649 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:07.516870 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.518009 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.519169 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.519970 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.520082 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.520133 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.520233 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.520283 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260810 23:07:07.520542 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.516268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41166 (local address 127.23.56.66:38303)
0810 23:07:07.516789 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.516793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.516801 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:07.516843 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.516845 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.516860 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.516950 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:07.517076 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.517736 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.519256 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.519983 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.520047 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.520104 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.520411 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.520462 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":93,"threads_started":1}
I20260810 23:07:07.520365 11169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.515972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:38511 (local address 127.0.0.1:41978)
0810 23:07:07.516159 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.516196 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.516267 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.516706 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.516709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.516711 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:07.516777 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.516779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.518144 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.518145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.519065 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.519067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.520090 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.520091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.520147 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.520151 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:07.520205 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 23:07:07.520242 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":6,"thread_start_us":60,"threads_started":1}
I20260810 23:07:07.520861 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.516070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47864 (local address 127.23.56.67:34205)
0810 23:07:07.516573 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.516578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.516588 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:07.516623 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.516626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.516641 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.516853 (+ 212us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:07.516944 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.518004 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.519329 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.519841 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.519932 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.520016 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.520735 (+ 719us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.520784 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":80,"threads_started":1}
I20260810 23:07:07.520437 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.516015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:38303 (local address 127.0.0.1:41166)
0810 23:07:07.516456 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.516465 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.516511 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.516947 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.516948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.516950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:07.517016 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.517018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.517846 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.517847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.519179 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.519182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.520140 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.520141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.520222 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.520227 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:07:07.520280 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 23:07:07.520323 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":29,"thread_start_us":94,"threads_started":1}
I20260810 23:07:07.521387 10770 tablet_service.cc:1467] Processing CreateTablet for tablet f927ff4e79c44dd38a52fd52e93ffb97 (DEFAULT_TABLE table=test-table [id=ed8eec898bd94ed2bc2b10cd7a7b8671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:07.521466 11073 tablet_service.cc:1467] Processing CreateTablet for tablet f927ff4e79c44dd38a52fd52e93ffb97 (DEFAULT_TABLE table=test-table [id=ed8eec898bd94ed2bc2b10cd7a7b8671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:07.521588 10917 tablet_service.cc:1467] Processing CreateTablet for tablet f927ff4e79c44dd38a52fd52e93ffb97 (DEFAULT_TABLE table=test-table [id=ed8eec898bd94ed2bc2b10cd7a7b8671]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:07.521714 11073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f927ff4e79c44dd38a52fd52e93ffb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:07.521739 10770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f927ff4e79c44dd38a52fd52e93ffb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:07.521857 10917 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f927ff4e79c44dd38a52fd52e93ffb97. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:07.524380 11174 tablet_bootstrap.cc:492] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: Bootstrap starting.
I20260810 23:07:07.524996 11174 tablet_bootstrap.cc:654] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:07.525036 11175 tablet_bootstrap.cc:492] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: Bootstrap starting.
I20260810 23:07:07.525641 11174 log.cc:824] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:07.525760 11175 tablet_bootstrap.cc:654] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:07.526322 11175 log.cc:824] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:07.526463 11174 tablet_bootstrap.cc:492] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: No bootstrap required, opened a new log
I20260810 23:07:07.526537 11174 ts_tablet_manager.cc:1397] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:07.526665 11176 tablet_bootstrap.cc:492] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4: Bootstrap starting.
I20260810 23:07:07.527024 11175 tablet_bootstrap.cc:492] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: No bootstrap required, opened a new log
I20260810 23:07:07.527107 11175 ts_tablet_manager.cc:1397] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:07.527292 11176 tablet_bootstrap.cc:654] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:07.527853 11176 log.cc:824] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:07.528703 11176 tablet_bootstrap.cc:492] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4: No bootstrap required, opened a new log
I20260810 23:07:07.528650 11174 raft_consensus.cc:348] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } }
I20260810 23:07:07.528810 11176 ts_tablet_manager.cc:1397] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:07.528826 11174 raft_consensus.cc:374] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:07.528899 11174 raft_consensus.cc:729] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf138f2f338b49558e599dedab07afcb, State: Initialized, Role: FOLLOWER
I20260810 23:07:07.529029 11174 consensus_queue.cc:260] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } }
I20260810 23:07:07.529027 11175 raft_consensus.cc:348] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } }
I20260810 23:07:07.529249 11175 raft_consensus.cc:374] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:07.529309 11175 raft_consensus.cc:729] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83d64078687347a9af21bbf08e916f9a, State: Initialized, Role: FOLLOWER
I20260810 23:07:07.529311 11174 ts_tablet_manager.cc:1428] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:07:07.529436 11154 heartbeater.cc:503] Master 127.23.56.126:33523 was elected leader, sending a full tablet report...
I20260810 23:07:07.529430 11175 consensus_queue.cc:260] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } }
I20260810 23:07:07.529776 11175 ts_tablet_manager.cc:1428] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260810 23:07:07.529896 10995 heartbeater.cc:503] Master 127.23.56.126:33523 was elected leader, sending a full tablet report...
I20260810 23:07:07.530857 11176 raft_consensus.cc:348] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } }
I20260810 23:07:07.531021 11176 raft_consensus.cc:374] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:07.531065 11176 raft_consensus.cc:729] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cafa637d7e9741619209bf926bd7a9e4, State: Initialized, Role: FOLLOWER
I20260810 23:07:07.531191 11176 consensus_queue.cc:260] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } }
I20260810 23:07:07.531478 11176 ts_tablet_manager.cc:1428] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:07:07.531514 10843 heartbeater.cc:503] Master 127.23.56.126:33523 was elected leader, sending a full tablet report...
I20260810 23:07:07.550699 11182 raft_consensus.cc:482] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:07:07.550796 11182 raft_consensus.cc:504] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } }
I20260810 23:07:07.551106 11182 leader_election.cc:288] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bf138f2f338b49558e599dedab07afcb (127.23.56.67:34205), 83d64078687347a9af21bbf08e916f9a (127.23.56.66:38303)
I20260810 23:07:07.553855 10845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.551353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:38303 (local address 127.23.56.65:38375)
0810 23:07:07.551422 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.551434 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.551482 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.551632 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.551633 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.551634 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:07.551703 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.551704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.552331 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.552332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.553050 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.553052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.553602 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.553603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.553686 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.553690 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:07.553737 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:07:07.553776 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260810 23:07:07.553946 11173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.551385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:38375 (local address 127.23.56.66:38303)
0810 23:07:07.551426 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.551428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.551466 (+ 38us) server_negotiation.cc:407] Connection header received
0810 23:07:07.551535 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.551535 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.551545 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.551612 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:07.551766 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.552242 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.553157 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.553510 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.553591 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.553601 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.553808 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.553854 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:07:07.554518 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f927ff4e79c44dd38a52fd52e93ffb97" candidate_uuid: "cafa637d7e9741619209bf926bd7a9e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d64078687347a9af21bbf08e916f9a" is_pre_election: true
I20260810 23:07:07.554672 10938 raft_consensus.cc:2454] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cafa637d7e9741619209bf926bd7a9e4 in term 0.
I20260810 23:07:07.554950 11170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.551403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:44165 (local address 127.23.56.67:34205)
0810 23:07:07.551452 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.551455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.551660 (+ 205us) server_negotiation.cc:407] Connection header received
0810 23:07:07.551729 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.551730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.551741 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.551803 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:07.552017 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.552543 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.553569 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.554238 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.554357 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.554368 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.554671 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.554771 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:07:07.555012 10719 leader_election.cc:302] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83d64078687347a9af21bbf08e916f9a, cafa637d7e9741619209bf926bd7a9e4; no voters:
I20260810 23:07:07.555330 11182 raft_consensus.cc:2781] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:07:07.555411 11182 raft_consensus.cc:482] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:07:07.555441 11182 raft_consensus.cc:3037] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:07.555796 11183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.551377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:34205 (local address 127.23.56.65:44165)
0810 23:07:07.551558 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.551569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.551667 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.551861 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.551862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.551863 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:07.551938 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.551939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.552657 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.552659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.553467 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.553469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.554397 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.554399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.554539 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.554544 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:07:07.555540 (+ 996us) client_negotiation.cc:240] Negotiation successful
0810 23:07:07.555625 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":111,"threads_started":1}
I20260810 23:07:07.556382 11182 raft_consensus.cc:504] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } }
I20260810 23:07:07.556550 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f927ff4e79c44dd38a52fd52e93ffb97" candidate_uuid: "cafa637d7e9741619209bf926bd7a9e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf138f2f338b49558e599dedab07afcb" is_pre_election: true
I20260810 23:07:07.556681 10938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f927ff4e79c44dd38a52fd52e93ffb97" candidate_uuid: "cafa637d7e9741619209bf926bd7a9e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83d64078687347a9af21bbf08e916f9a"
I20260810 23:07:07.556731 11096 raft_consensus.cc:2454] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cafa637d7e9741619209bf926bd7a9e4 in term 0.
I20260810 23:07:07.556759 10938 raft_consensus.cc:3037] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:07.556938 11096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f927ff4e79c44dd38a52fd52e93ffb97" candidate_uuid: "cafa637d7e9741619209bf926bd7a9e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf138f2f338b49558e599dedab07afcb"
I20260810 23:07:07.557018 11096 raft_consensus.cc:3037] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:07.557643 10938 raft_consensus.cc:2454] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cafa637d7e9741619209bf926bd7a9e4 in term 1.
I20260810 23:07:07.557747 11182 leader_election.cc:288] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [CANDIDATE]: Term 1 election: Requested vote from peers bf138f2f338b49558e599dedab07afcb (127.23.56.67:34205), 83d64078687347a9af21bbf08e916f9a (127.23.56.66:38303)
I20260810 23:07:07.557848 10719 leader_election.cc:302] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83d64078687347a9af21bbf08e916f9a, cafa637d7e9741619209bf926bd7a9e4; no voters:
I20260810 23:07:07.557843 11096 raft_consensus.cc:2454] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cafa637d7e9741619209bf926bd7a9e4 in term 1.
I20260810 23:07:07.557935 11182 raft_consensus.cc:2781] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:07.558095 11182 raft_consensus.cc:686] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [term 1 LEADER]: Becoming Leader. State: Replica: cafa637d7e9741619209bf926bd7a9e4, State: Running, Role: LEADER
I20260810 23:07:07.558202 11182 consensus_queue.cc:237] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } }
I20260810 23:07:07.559116 10620 catalog_manager.cc:5495] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to cafa637d7e9741619209bf926bd7a9e4 (127.23.56.65). New cstate: current_term: 1 leader_uuid: "cafa637d7e9741619209bf926bd7a9e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafa637d7e9741619209bf926bd7a9e4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 } health_report { overall_health: UNKNOWN } } }
I20260810 23:07:07.569789 11171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.567400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.23.56.65:38511)
0810 23:07:07.567462 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.567464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.567476 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:07:07.567520 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.567521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.567537 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.567603 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:07.567707 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.568219 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.568942 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.569121 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.569179 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.569389 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:07.569485 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:07.569606 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.569699 (+ 93us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.569718 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:07:07.571616 10938 raft_consensus.cc:1261] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [term 1 FOLLOWER]: Refusing update from remote peer cafa637d7e9741619209bf926bd7a9e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:07.571664 11096 raft_consensus.cc:1261] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [term 1 FOLLOWER]: Refusing update from remote peer cafa637d7e9741619209bf926bd7a9e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:07.572016 11184 consensus_queue.cc:1035] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83d64078687347a9af21bbf08e916f9a" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 38303 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:07.572134 11182 consensus_queue.cc:1035] T f927ff4e79c44dd38a52fd52e93ffb97 P cafa637d7e9741619209bf926bd7a9e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf138f2f338b49558e599dedab07afcb" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 34205 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:07.574488 11187 mvcc.cc:205] Tried to move back new op lower bound from 7317107620130402304 to 7317107620079136768. Current Snapshot: MvccSnapshot[applied={T|T < 7317107620130402304}]
I20260810 23:07:07.574755 11189 mvcc.cc:205] Tried to move back new op lower bound from 7317107620130402304 to 7317107620079136768. Current Snapshot: MvccSnapshot[applied={T|T < 7317107620130402304}]
I20260810 23:07:07.575089 11188 mvcc.cc:205] Tried to move back new op lower bound from 7317107620130402304 to 7317107620079136768. Current Snapshot: MvccSnapshot[applied={T|T < 7317107620130402304}]
I20260810 23:07:07.576841 10621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:07.576953 10621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:07.577849 10621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2442987a38194af7b450656a6d946709: Sending DeleteTablet for 3 replicas of tablet f927ff4e79c44dd38a52fd52e93ffb97
I20260810 23:07:07.578233 10770 tablet_service.cc:1514] Processing DeleteTablet for tablet f927ff4e79c44dd38a52fd52e93ffb97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:07 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:41978
I20260810 23:07:07.578217 10917 tablet_service.cc:1514] Processing DeleteTablet for tablet f927ff4e79c44dd38a52fd52e93ffb97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:07 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:41166
I20260810 23:07:07.578469 11073 tablet_service.cc:1514] Processing DeleteTablet for tablet f927ff4e79c44dd38a52fd52e93ffb97 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:07 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:47864
I20260810 23:07:07.578598 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 10698
I20260810 23:07:07.578724 11200 tablet_replica.cc:330] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: stopping tablet replica
I20260810 23:07:07.578858 11200 raft_consensus.cc:2229] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:07.579031 11200 raft_consensus.cc:2258] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:07.578696 11198 tablet_replica.cc:330] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: stopping tablet replica
I20260810 23:07:07.580364 11198 raft_consensus.cc:2229] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:07.580479 11198 raft_consensus.cc:2258] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:07.580816 11200 ts_tablet_manager.cc:1905] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:07.580868 11198 ts_tablet_manager.cc:1905] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:07.583562 11198 ts_tablet_manager.cc:1918] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:07.583647 11198 log.cc:1196] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/f927ff4e79c44dd38a52fd52e93ffb97
I20260810 23:07:07.583971 11198 ts_tablet_manager.cc:1939] T f927ff4e79c44dd38a52fd52e93ffb97 P 83d64078687347a9af21bbf08e916f9a: Deleting consensus metadata
I20260810 23:07:07.584430 10605 catalog_manager.cc:4841] TS 83d64078687347a9af21bbf08e916f9a (127.23.56.66:38303): tablet f927ff4e79c44dd38a52fd52e93ffb97 (table test-table [id=ed8eec898bd94ed2bc2b10cd7a7b8671]) successfully deleted
W20260810 23:07:07.588711 10603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:38511 (error 108)
I20260810 23:07:07.588943 11200 ts_tablet_manager.cc:1918] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:07.589035 11200 log.cc:1196] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/f927ff4e79c44dd38a52fd52e93ffb97
I20260810 23:07:07.589037 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 10848
I20260810 23:07:07.589340 11200 ts_tablet_manager.cc:1939] T f927ff4e79c44dd38a52fd52e93ffb97 P bf138f2f338b49558e599dedab07afcb: Deleting consensus metadata
I20260810 23:07:07.592551 11167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.592220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:38511 (local address 127.0.0.1:41982)
0810 23:07:07.592466 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.592510 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:38511: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260810 23:07:07.592716 10606 catalog_manager.cc:4841] TS bf138f2f338b49558e599dedab07afcb (127.23.56.67:34205): tablet f927ff4e79c44dd38a52fd52e93ffb97 (table test-table [id=ed8eec898bd94ed2bc2b10cd7a7b8671]) successfully deleted
W20260810 23:07:07.593482 10603 catalog_manager.cc:4584] TS cafa637d7e9741619209bf926bd7a9e4 (127.23.56.65:38511): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f927ff4e79c44dd38a52fd52e93ffb97: Network error: Client connection negotiation failed: client connection to 127.23.56.65:38511: connect: Connection refused (error 111)
I20260810 23:07:07.599362 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 11000
I20260810 23:07:07.608541 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 10586
2026-08-10T23:07:07Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2039 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T23:07:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:07Z Disabled control of system clock
I20260810 23:07:07.641403 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45713
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:45713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:07:07.721946 11206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:07.722187 11206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:07.722239 11206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:07.723417 11206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:07.723483 11206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:07.723505 11206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:07.723527 11206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:07.723557 11206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:07.724913 11206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:45713
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:45713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:07.725190 11206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:07.725485 11206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:07.730319 11215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:07.730342 11213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:07.730352 11212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:07.730595 11206 server_base.cc:1034] running on GCE node
I20260810 23:07:07.730818 11206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:07.731155 11206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:07.732311 11206 hybrid_clock.cc:648] HybridClock initialized: now 1786403227732291 us; error 37 us; skew 500 ppm
I20260810 23:07:07.733544 11206 webserver.cc:460] Webserver started at http://127.23.56.126:36261/ using document root <none> and password file <none>
I20260810 23:07:07.733783 11206 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:07.733851 11206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:07.734018 11206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:07.735078 11206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "180f9f21c4384e8ba8d5b8de16860c62"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "79da4d11329dde0130b4b9888e4d9aaa"
server_key_iv: "754acfc2068bf97578b529fc83840f38"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.735517 11206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "180f9f21c4384e8ba8d5b8de16860c62"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "79da4d11329dde0130b4b9888e4d9aaa"
server_key_iv: "754acfc2068bf97578b529fc83840f38"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.736804 11206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:07.737566 11222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.737744 11206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:07.737852 11206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "180f9f21c4384e8ba8d5b8de16860c62"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "79da4d11329dde0130b4b9888e4d9aaa"
server_key_iv: "754acfc2068bf97578b529fc83840f38"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.737937 11206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:07.756076 11206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:07.756397 11206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:07.756554 11206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:07.760533 11206 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:45713
I20260810 23:07:07.760576 11280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:45713 every 8 connection(s)
I20260810 23:07:07.760949 11206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:07.761902 11281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:07.764180 11281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62: Bootstrap starting.
I20260810 23:07:07.764791 11281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:07.765056 11281 log.cc:824] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:07.765734 11281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62: No bootstrap required, opened a new log
I20260810 23:07:07.765740 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 11206
I20260810 23:07:07.765857 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:07.766045 23777 external_mini_cluster.cc:1428] Setting key 53f0673b18b7f42b1a9e93a2a467b080
I20260810 23:07:07.767712 11281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180f9f21c4384e8ba8d5b8de16860c62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45713 } }
I20260810 23:07:07.767844 11281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:07.767866 11281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 180f9f21c4384e8ba8d5b8de16860c62, State: Initialized, Role: FOLLOWER
I20260810 23:07:07.768009 11281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180f9f21c4384e8ba8d5b8de16860c62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45713 } }
I20260810 23:07:07.768095 11281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:07.768132 11281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:07.768182 11281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:07.768705 11281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180f9f21c4384e8ba8d5b8de16860c62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45713 } }
I20260810 23:07:07.768790 11281 leader_election.cc:302] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 180f9f21c4384e8ba8d5b8de16860c62; no voters:
I20260810 23:07:07.769026 11281 leader_election.cc:288] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:07.769053 11286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:07.769167 11286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [term 1 LEADER]: Becoming Leader. State: Replica: 180f9f21c4384e8ba8d5b8de16860c62, State: Running, Role: LEADER
I20260810 23:07:07.769248 11286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180f9f21c4384e8ba8d5b8de16860c62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45713 } }
I20260810 23:07:07.769281 11281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:07.769572 11286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "180f9f21c4384e8ba8d5b8de16860c62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180f9f21c4384e8ba8d5b8de16860c62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45713 } } }
I20260810 23:07:07.769629 11287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 180f9f21c4384e8ba8d5b8de16860c62. Latest consensus state: current_term: 1 leader_uuid: "180f9f21c4384e8ba8d5b8de16860c62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "180f9f21c4384e8ba8d5b8de16860c62" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 45713 } } }
I20260810 23:07:07.769647 11286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:07.769716 11287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:07.769927 11292 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:07.770396 11292 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:07.771881 11292 catalog_manager.cc:1347] Generated new cluster ID: e8e6ddb4346d408692f1ac794365a78f
I20260810 23:07:07.771934 11292 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:07.773375 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.766617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47452 (local address 127.23.56.126:45713)
0810 23:07:07.766812 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.766821 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.766850 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:07:07.767704 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.767711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.767784 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.767959 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:07.768294 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.769969 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.771243 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.771789 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.772262 (+ 473us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.772653 (+ 391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:07.772657 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:07.772657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:07.772872 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:07.772921 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:07.772934 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:07.773000 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.773103 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.773177 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260810 23:07:07.777923 11292 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:07.778600 11292 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:07.787778 11292 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62: Generated new TSK 0
I20260810 23:07:07.787963 11292 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:07.790719 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45713
--builtin_ntp_servers=127.23.56.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:07:07.873924 11308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:07.874116 11308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:07.874137 11308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:07.875305 11308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:07.875344 11308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:07.875368 11308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:07:07.876770 11308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:07.876999 11308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:07.877321 11308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:07:07.882305 11308 server_base.cc:1034] running on GCE node
W20260810 23:07:07.882275 11314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:07.882453 11313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:07.882264 11316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:07.882678 11308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:07.882937 11308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:07.884088 11308 hybrid_clock.cc:648] HybridClock initialized: now 1786403227884079 us; error 26 us; skew 500 ppm
I20260810 23:07:07.885378 11308 webserver.cc:460] Webserver started at http://127.23.56.65:43911/ using document root <none> and password file <none>
I20260810 23:07:07.885653 11308 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:07.885699 11308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:07.885836 11308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:07.886897 11308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "35bc9a2205e94bfa9ef8941daff3c5a4"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "e604cb15f883ef2f324ab32be872d8c2"
server_key_iv: "ed238f92ccdb033afa101487c79f7f6b"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.887288 11308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "35bc9a2205e94bfa9ef8941daff3c5a4"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "e604cb15f883ef2f324ab32be872d8c2"
server_key_iv: "ed238f92ccdb033afa101487c79f7f6b"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.888624 11308 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:07:07.889426 11322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.889611 11308 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:07:07.889708 11308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "35bc9a2205e94bfa9ef8941daff3c5a4"
format_stamp: "Formatted at 2026-08-10 23:07:07 on dist-test-slave-1xrh"
server_key: "e604cb15f883ef2f324ab32be872d8c2"
server_key_iv: "ed238f92ccdb033afa101487c79f7f6b"
server_key_version: "encryptionkey@0"
I20260810 23:07:07.889786 11308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:07.902917 11308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:07.903285 11308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:07.903439 11308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:07.903657 11308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:07.903987 11308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:07.904043 11308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.904084 11308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:07.904110 11308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:07.909965 11308 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:38209
I20260810 23:07:07.910013 11449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:38209 every 8 connection(s)
I20260810 23:07:07.910320 11308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:07:07.915416 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.910980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:44871 (local address 127.23.56.126:45713)
0810 23:07:07.911028 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:07.911030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:07.911324 (+ 294us) server_negotiation.cc:407] Connection header received
0810 23:07:07.911866 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:07.911868 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:07.911883 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:07.911948 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:07.912604 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.913098 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.914033 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.914224 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.914300 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.915060 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:07.915062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:07.915062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:07.915067 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:07.915081 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:07.915086 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:07.915180 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:07.915299 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 23:07:07.915330 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:07:07.915613 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:07.910957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45713 (local address 127.23.56.65:44871)
0810 23:07:07.911200 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:07.911239 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:07.911738 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:07.912017 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:07.912021 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:07.912030 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:07.912508 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.912513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.913198 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.913199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.913942 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:07.913944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:07.914307 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:07.914309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:07.914441 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:07.914924 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:07.914925 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:07.914951 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:07.914971 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:07.915184 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:07.915190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:07.915193 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:07:07.915299 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 23:07:07.915395 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260810 23:07:07.916107 11450 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45713
I20260810 23:07:07.916180 11450 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:07.916308 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 11308
I20260810 23:07:07.916365 11450 heartbeater.cc:511] Master 127.23.56.126:45713 requested a full tablet report, sending...
I20260810 23:07:07.916410 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:07:07.916597 23777 external_mini_cluster.cc:1428] Setting key cc2ee13fd2a9c50518609901c258f2e8
I20260810 23:07:07.916944 11241 ts_manager.cc:194] Registered new tserver with Master: 35bc9a2205e94bfa9ef8941daff3c5a4 (127.23.56.65:38209)
I20260810 23:07:07.917726 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45713
--builtin_ntp_servers=127.23.56.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:07:07.917744 11241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.65:44871
W20260810 23:07:07.999795 11454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:07.999989 11454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:08.000027 11454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:08.001207 11454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:08.001248 11454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:08.001273 11454 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:07:08.002691 11454 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:08.002935 11454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:08.003250 11454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:08.008369 11460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:08.008368 11464 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:08.008368 11462 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:08.008603 11454 server_base.cc:1034] running on GCE node
I20260810 23:07:08.008833 11454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:08.009075 11454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:08.010237 11454 hybrid_clock.cc:648] HybridClock initialized: now 1786403228010233 us; error 36 us; skew 500 ppm
I20260810 23:07:08.011596 11454 webserver.cc:460] Webserver started at http://127.23.56.66:44243/ using document root <none> and password file <none>
I20260810 23:07:08.011858 11454 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:08.011937 11454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:08.012087 11454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:08.013216 11454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "2d10e75f7ce44f46a39c3719153077a5"
format_stamp: "Formatted at 2026-08-10 23:07:08 on dist-test-slave-1xrh"
server_key: "76400ed2bcb1473407dbc68e06cec365"
server_key_iv: "2d7ffcd930fff622f4a55df219c9ea28"
server_key_version: "encryptionkey@0"
I20260810 23:07:08.013579 11454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "2d10e75f7ce44f46a39c3719153077a5"
format_stamp: "Formatted at 2026-08-10 23:07:08 on dist-test-slave-1xrh"
server_key: "76400ed2bcb1473407dbc68e06cec365"
server_key_iv: "2d7ffcd930fff622f4a55df219c9ea28"
server_key_version: "encryptionkey@0"
I20260810 23:07:08.014863 11454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:08.015645 11470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:08.015821 11454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:08.015935 11454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "2d10e75f7ce44f46a39c3719153077a5"
format_stamp: "Formatted at 2026-08-10 23:07:08 on dist-test-slave-1xrh"
server_key: "76400ed2bcb1473407dbc68e06cec365"
server_key_iv: "2d7ffcd930fff622f4a55df219c9ea28"
server_key_version: "encryptionkey@0"
I20260810 23:07:08.016017 11454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:08.026084 11454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:08.026353 11454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:08.026504 11454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:08.026710 11454 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:08.027019 11454 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:08.027074 11454 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:08.027114 11454 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:08.027179 11454 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:08.033012 11454 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:35909
I20260810 23:07:08.033084 11595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:35909 every 8 connection(s)
I20260810 23:07:08.033370 11454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:07:08.038221 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.034140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:46695 (local address 127.23.56.126:45713)
0810 23:07:08.034187 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.034189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.034373 (+ 184us) server_negotiation.cc:407] Connection header received
0810 23:07:08.034825 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.034826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.034836 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.034906 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:08.035550 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.036029 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.036982 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.037172 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.037223 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.038023 (+ 800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:08.038024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:08.038024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:08.038028 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:08.038040 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:08.038043 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:08.038080 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:08.038144 (+ 64us) server_negotiation.cc:299] Negotiation successful
0810 23:07:08.038164 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:07:08.038424 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.034079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45713 (local address 127.23.56.66:46695)
0810 23:07:08.034268 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.034306 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:08.034744 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:08.034958 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:08.034960 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:08.034968 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:08.035478 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.035482 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.036128 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.036129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.036904 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.036906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.037248 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.037249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.037349 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.037898 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:08.037899 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:08.037927 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:08.037949 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:08.038082 (+ 133us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:08.038084 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:08.038086 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:07:08.038131 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 23:07:08.038202 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":98,"threads_started":1}
I20260810 23:07:08.038975 11597 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45713
I20260810 23:07:08.039050 11597 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:08.039254 11597 heartbeater.cc:511] Master 127.23.56.126:45713 requested a full tablet report, sending...
I20260810 23:07:08.039618 11241 ts_manager.cc:194] Registered new tserver with Master: 2d10e75f7ce44f46a39c3719153077a5 (127.23.56.66:35909)
I20260810 23:07:08.039983 11241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.66:46695
I20260810 23:07:08.043318 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 11454
I20260810 23:07:08.043433 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:07:08.043630 23777 external_mini_cluster.cc:1428] Setting key 5c6a24f8969b6d1e2df1eca42ce4e94f
I20260810 23:07:08.044541 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45713
--builtin_ntp_servers=127.23.56.84:38103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:07:08.129894 11600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:08.130070 11600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:08.130090 11600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:08.131229 11600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:08.131268 11600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:08.131292 11600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:07:08.132666 11600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:38103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:45713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:08.132905 11600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:08.133190 11600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:08.138188 11606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:08.138222 11609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:08.138218 11600 server_base.cc:1034] running on GCE node
W20260810 23:07:08.138178 11607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:08.138612 11600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:08.138809 11600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:08.139959 11600 hybrid_clock.cc:648] HybridClock initialized: now 1786403228139933 us; error 45 us; skew 500 ppm
I20260810 23:07:08.141105 11600 webserver.cc:460] Webserver started at http://127.23.56.67:41389/ using document root <none> and password file <none>
I20260810 23:07:08.141286 11600 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:08.141325 11600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:08.141472 11600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:08.142539 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "48889e06937a4c4b96436953be28cbd4"
format_stamp: "Formatted at 2026-08-10 23:07:08 on dist-test-slave-1xrh"
server_key: "7d2828690063ca5def3af943376ec7d1"
server_key_iv: "58df4fd451559ebc5f06f5929b8dca2f"
server_key_version: "encryptionkey@0"
I20260810 23:07:08.142972 11600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "48889e06937a4c4b96436953be28cbd4"
format_stamp: "Formatted at 2026-08-10 23:07:08 on dist-test-slave-1xrh"
server_key: "7d2828690063ca5def3af943376ec7d1"
server_key_iv: "58df4fd451559ebc5f06f5929b8dca2f"
server_key_version: "encryptionkey@0"
I20260810 23:07:08.144286 11600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:08.144927 11616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:08.145092 11600 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:08.145171 11600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "48889e06937a4c4b96436953be28cbd4"
format_stamp: "Formatted at 2026-08-10 23:07:08 on dist-test-slave-1xrh"
server_key: "7d2828690063ca5def3af943376ec7d1"
server_key_iv: "58df4fd451559ebc5f06f5929b8dca2f"
server_key_version: "encryptionkey@0"
I20260810 23:07:08.145222 11600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:08.156860 11600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:08.157083 11600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:08.157176 11600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:08.157344 11600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:08.157608 11600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:08.157639 11600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:08.157658 11600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:08.157670 11600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:08.163082 11600 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:41179
I20260810 23:07:08.163173 11741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:41179 every 8 connection(s)
I20260810 23:07:08.163514 11600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:07:08.168967 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.164306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:49675 (local address 127.23.56.126:45713)
0810 23:07:08.164352 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.164354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.164617 (+ 263us) server_negotiation.cc:407] Connection header received
0810 23:07:08.165264 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.165266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.165279 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.165332 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:08.166023 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.166528 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.167589 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.167785 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.167860 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.168649 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:08.168651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:08.168652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:08.168657 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:08.168673 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:08.168677 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:08.168746 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:08.168850 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 23:07:08.168882 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:07:08.169209 11745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.164278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:45713 (local address 127.23.56.67:49675)
0810 23:07:08.164496 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.164529 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:08.165155 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:08.165399 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:08.165404 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:08.165413 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:08.165926 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.165930 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.166627 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.166628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.167490 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.167492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.167868 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.167869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.168004 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.168508 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:08.168509 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:08.168536 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:08.168559 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:08.168748 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:08.168755 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:08.168758 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:07:08.168837 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 23:07:08.168971 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":106,"threads_started":1}
I20260810 23:07:08.169736 11742 heartbeater.cc:347] Connected to a master server at 127.23.56.126:45713
I20260810 23:07:08.169858 11742 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:08.169988 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 11600
I20260810 23:07:08.170075 11742 heartbeater.cc:511] Master 127.23.56.126:45713 requested a full tablet report, sending...
I20260810 23:07:08.170089 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:07:08.170320 23777 external_mini_cluster.cc:1428] Setting key 570202432a49e077c510d3691d44edfb
I20260810 23:07:08.170519 11241 ts_manager.cc:194] Registered new tserver with Master: 48889e06937a4c4b96436953be28cbd4 (127.23.56.67:41179)
I20260810 23:07:08.171038 11241 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.67:49675
I20260810 23:07:08.171274 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:07:08.175264 11285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.172671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.23.56.126:45713)
0810 23:07:08.172709 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.172711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.172826 (+ 115us) server_negotiation.cc:407] Connection header received
0810 23:07:08.172881 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.172881 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.172890 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.172939 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:08.173085 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.173521 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.174211 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.174386 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.174433 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.175049 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:08.175050 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:08.175050 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:08.175054 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:08.175066 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:08.175070 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:08.175112 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:08.175185 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 23:07:08.175213 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:07:08.176690 11241 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:07:08.178241 11241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:07:08.184347 11753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.180437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:38209 (local address 127.0.0.1:49946)
0810 23:07:08.180839 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.180848 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:08.180890 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:08.181146 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:08.181147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:08.181149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:08.181191 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.181193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.182087 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.182089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.183163 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.183165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.184062 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.184063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.184138 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.184151 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 23:07:08.184225 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:07:08.184278 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":115,"threads_started":1}
I20260810 23:07:08.184360 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.180797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.23.56.65:38209)
0810 23:07:08.181002 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.181006 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.181016 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:08.181055 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.181057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.181074 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.181151 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:08.181249 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.181997 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.183259 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.183967 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.184062 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.184155 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:08.184240 (+ 85us) server_negotiation.cc:299] Negotiation successful
0810 23:07:08.184277 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
I20260810 23:07:08.184827 11752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.180431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:35909 (local address 127.0.0.1:35642)
0810 23:07:08.180623 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.180663 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:08.180731 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:08.180992 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:08.180995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:08.180997 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:08.181307 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.181309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.182632 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.182633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.183556 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.183559 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.184577 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.184578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.184645 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.184649 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:08.184702 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 23:07:08.184749 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260810 23:07:08.185070 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.180612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.23.56.66:35909)
0810 23:07:08.180789 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.180795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.180808 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:07:08.180850 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.180851 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.180867 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.180981 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:08.181399 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.182527 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.183640 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.184391 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.184462 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.184542 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:08.184930 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 23:07:08.184979 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260810 23:07:08.185384 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.180667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.23.56.67:41179)
0810 23:07:08.180825 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.180828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.181059 (+ 231us) server_negotiation.cc:407] Connection header received
0810 23:07:08.181134 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.181136 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.181150 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.181215 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:08.181401 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.182518 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.183765 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.184687 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.184745 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.184819 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:08.185028 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 23:07:08.185068 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":75,"threads_started":1}
I20260810 23:07:08.185286 11376 tablet_service.cc:1467] Processing CreateTablet for tablet e94e68b6a9c74899bbba3667a1bcbf3a (DEFAULT_TABLE table=test-table [id=873ca5b7b12a45039c14c6112848ba4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:08.185613 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.180449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41179 (local address 127.0.0.1:35724)
0810 23:07:08.180995 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.181005 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:08.181063 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:08.181276 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:08.181278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:08.181281 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:08.181329 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.181331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.182611 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.182612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.183680 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.183682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.184867 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.184868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.184930 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.184934 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:08.185486 (+ 552us) client_negotiation.cc:240] Negotiation successful
0810 23:07:08.185539 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":95,"threads_started":1}
I20260810 23:07:08.185688 11376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94e68b6a9c74899bbba3667a1bcbf3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:08.185722 11524 tablet_service.cc:1467] Processing CreateTablet for tablet e94e68b6a9c74899bbba3667a1bcbf3a (DEFAULT_TABLE table=test-table [id=873ca5b7b12a45039c14c6112848ba4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:08.186053 11524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94e68b6a9c74899bbba3667a1bcbf3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:08.187729 11670 tablet_service.cc:1467] Processing CreateTablet for tablet e94e68b6a9c74899bbba3667a1bcbf3a (DEFAULT_TABLE table=test-table [id=873ca5b7b12a45039c14c6112848ba4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:08.188067 11670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94e68b6a9c74899bbba3667a1bcbf3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:08.189222 11757 tablet_bootstrap.cc:492] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4: Bootstrap starting.
I20260810 23:07:08.189446 11758 tablet_bootstrap.cc:492] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: Bootstrap starting.
I20260810 23:07:08.190132 11758 tablet_bootstrap.cc:654] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:08.190467 11758 log.cc:824] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:08.191246 11760 tablet_bootstrap.cc:492] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: Bootstrap starting.
I20260810 23:07:08.191294 11758 tablet_bootstrap.cc:492] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: No bootstrap required, opened a new log
I20260810 23:07:08.191358 11758 ts_tablet_manager.cc:1397] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:08.191370 11757 tablet_bootstrap.cc:654] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:08.191699 11757 log.cc:824] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:08.192020 11760 tablet_bootstrap.cc:654] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:08.192406 11760 log.cc:824] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:08.192453 11757 tablet_bootstrap.cc:492] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4: No bootstrap required, opened a new log
I20260810 23:07:08.192562 11757 ts_tablet_manager.cc:1397] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:07:08.193115 11760 tablet_bootstrap.cc:492] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: No bootstrap required, opened a new log
I20260810 23:07:08.193193 11760 ts_tablet_manager.cc:1397] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260810 23:07:08.193694 11758 raft_consensus.cc:348] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } }
I20260810 23:07:08.193821 11758 raft_consensus.cc:374] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:08.193841 11758 raft_consensus.cc:729] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d10e75f7ce44f46a39c3719153077a5, State: Initialized, Role: FOLLOWER
I20260810 23:07:08.193955 11758 consensus_queue.cc:260] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } }
I20260810 23:07:08.194285 11597 heartbeater.cc:503] Master 127.23.56.126:45713 was elected leader, sending a full tablet report...
I20260810 23:07:08.194270 11758 ts_tablet_manager.cc:1428] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:07:08.195299 11757 raft_consensus.cc:348] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } }
I20260810 23:07:08.195451 11757 raft_consensus.cc:374] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:08.195485 11757 raft_consensus.cc:729] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35bc9a2205e94bfa9ef8941daff3c5a4, State: Initialized, Role: FOLLOWER
I20260810 23:07:08.195603 11757 consensus_queue.cc:260] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } }
I20260810 23:07:08.195858 11757 ts_tablet_manager.cc:1428] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260810 23:07:08.195899 11450 heartbeater.cc:503] Master 127.23.56.126:45713 was elected leader, sending a full tablet report...
I20260810 23:07:08.195847 11760 raft_consensus.cc:348] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } }
I20260810 23:07:08.195983 11760 raft_consensus.cc:374] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:08.196022 11760 raft_consensus.cc:729] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48889e06937a4c4b96436953be28cbd4, State: Initialized, Role: FOLLOWER
I20260810 23:07:08.196134 11760 consensus_queue.cc:260] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } }
I20260810 23:07:08.196375 11760 ts_tablet_manager.cc:1428] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 23:07:08.196408 11742 heartbeater.cc:503] Master 127.23.56.126:45713 was elected leader, sending a full tablet report...
W20260810 23:07:08.283922 11598 tablet.cc:2369] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:07:08.300168 11765 raft_consensus.cc:482] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:07:08.300324 11765 raft_consensus.cc:504] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } }
I20260810 23:07:08.300715 11765 leader_election.cc:288] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2d10e75f7ce44f46a39c3719153077a5 (127.23.56.66:35909), 48889e06937a4c4b96436953be28cbd4 (127.23.56.67:41179)
I20260810 23:07:08.303735 11754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.301055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:43441 (local address 127.23.56.66:35909)
0810 23:07:08.301101 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.301103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.301113 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:08.301148 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.301149 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.301159 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.301218 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:08.301408 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.301924 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.302971 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.303366 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.303430 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.303437 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:08.303605 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 23:07:08.303654 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:07:08.303761 11755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.301027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:41101 (local address 127.23.56.67:41179)
0810 23:07:08.301096 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.301098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.301209 (+ 111us) server_negotiation.cc:407] Connection header received
0810 23:07:08.301264 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.301265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.301275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.301331 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:08.301518 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.301987 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.302933 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.303410 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.303483 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.303491 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:08.303661 (+ 170us) server_negotiation.cc:299] Negotiation successful
0810 23:07:08.303700 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:07:08.303747 11453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.300905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:35909 (local address 127.23.56.65:43441)
0810 23:07:08.300972 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.300986 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:08.301041 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:08.301230 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:08.301231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:08.301233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:08.301318 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.301320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.302005 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.302006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.302847 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.302850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.303447 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.303448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.303519 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.303523 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:08.303581 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 23:07:08.303655 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":35}
I20260810 23:07:08.303767 11767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.300912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41179 (local address 127.23.56.65:41101)
0810 23:07:08.301128 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.301138 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:08.301201 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:08.301346 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:08.301347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:08.301366 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:08.301440 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.301442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.302069 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.302070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.302831 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:08.302834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.303505 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.303506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:08.303578 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.303582 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:08.303639 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 23:07:08.303673 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":23,"thread_start_us":113,"threads_started":1}
I20260810 23:07:08.304256 11546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94e68b6a9c74899bbba3667a1bcbf3a" candidate_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d10e75f7ce44f46a39c3719153077a5" is_pre_election: true
I20260810 23:07:08.304337 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94e68b6a9c74899bbba3667a1bcbf3a" candidate_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48889e06937a4c4b96436953be28cbd4" is_pre_election: true
I20260810 23:07:08.304411 11546 raft_consensus.cc:2454] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35bc9a2205e94bfa9ef8941daff3c5a4 in term 0.
I20260810 23:07:08.304500 11690 raft_consensus.cc:2454] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35bc9a2205e94bfa9ef8941daff3c5a4 in term 0.
I20260810 23:07:08.304682 11325 leader_election.cc:302] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d10e75f7ce44f46a39c3719153077a5, 35bc9a2205e94bfa9ef8941daff3c5a4; no voters:
I20260810 23:07:08.304837 11765 raft_consensus.cc:2781] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:07:08.304914 11765 raft_consensus.cc:482] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:07:08.304948 11765 raft_consensus.cc:3037] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:08.305665 11765 raft_consensus.cc:504] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } }
I20260810 23:07:08.305814 11765 leader_election.cc:288] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [CANDIDATE]: Term 1 election: Requested vote from peers 2d10e75f7ce44f46a39c3719153077a5 (127.23.56.66:35909), 48889e06937a4c4b96436953be28cbd4 (127.23.56.67:41179)
I20260810 23:07:08.305964 11690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94e68b6a9c74899bbba3667a1bcbf3a" candidate_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "48889e06937a4c4b96436953be28cbd4"
I20260810 23:07:08.305979 11546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94e68b6a9c74899bbba3667a1bcbf3a" candidate_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d10e75f7ce44f46a39c3719153077a5"
I20260810 23:07:08.306041 11690 raft_consensus.cc:3037] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:08.306073 11546 raft_consensus.cc:3037] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:08.306737 11690 raft_consensus.cc:2454] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35bc9a2205e94bfa9ef8941daff3c5a4 in term 1.
I20260810 23:07:08.306917 11546 raft_consensus.cc:2454] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35bc9a2205e94bfa9ef8941daff3c5a4 in term 1.
I20260810 23:07:08.306916 11324 leader_election.cc:302] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35bc9a2205e94bfa9ef8941daff3c5a4, 48889e06937a4c4b96436953be28cbd4; no voters:
I20260810 23:07:08.306998 11765 raft_consensus.cc:2781] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:08.307173 11765 raft_consensus.cc:686] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 1 LEADER]: Becoming Leader. State: Replica: 35bc9a2205e94bfa9ef8941daff3c5a4, State: Running, Role: LEADER
I20260810 23:07:08.307269 11765 consensus_queue.cc:237] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } }
I20260810 23:07:08.308094 11238 catalog_manager.cc:5495] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35bc9a2205e94bfa9ef8941daff3c5a4 (127.23.56.65). New cstate: current_term: 1 leader_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35bc9a2205e94bfa9ef8941daff3c5a4" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 38209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 } health_report { overall_health: UNKNOWN } } }
I20260810 23:07:08.332903 11756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.330639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49958 (local address 127.23.56.65:38209)
0810 23:07:08.330730 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:08.330732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:08.330739 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:07:08.330777 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:08.330778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:08.330797 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:08.330856 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:08.330982 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.331472 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.332168 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:08.332341 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:08.332393 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:08.332542 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:08.332637 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:08.332696 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:08.332744 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 23:07:08.332760 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 23:07:08.334806 11546 raft_consensus.cc:1261] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [term 1 FOLLOWER]: Refusing update from remote peer 35bc9a2205e94bfa9ef8941daff3c5a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:08.334806 11690 raft_consensus.cc:1261] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [term 1 FOLLOWER]: Refusing update from remote peer 35bc9a2205e94bfa9ef8941daff3c5a4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:08.335204 11768 consensus_queue.cc:1035] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d10e75f7ce44f46a39c3719153077a5" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 35909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:08.335361 11765 consensus_queue.cc:1035] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "48889e06937a4c4b96436953be28cbd4" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:08.337742 11771 mvcc.cc:205] Tried to move back new op lower bound from 7317107623256535040 to 7317107623147151360. Current Snapshot: MvccSnapshot[applied={T|T < 7317107623256535040}]
I20260810 23:07:08.338451 11772 mvcc.cc:205] Tried to move back new op lower bound from 7317107623256535040 to 7317107623147151360. Current Snapshot: MvccSnapshot[applied={T|T < 7317107623256535040}]
I20260810 23:07:08.341049 11238 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:08.341142 11238 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:08.341922 11238 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 180f9f21c4384e8ba8d5b8de16860c62: Sending DeleteTablet for 3 replicas of tablet e94e68b6a9c74899bbba3667a1bcbf3a
I20260810 23:07:08.342273 11670 tablet_service.cc:1514] Processing DeleteTablet for tablet e94e68b6a9c74899bbba3667a1bcbf3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:08 UTC) from {username='slave'} at 127.0.0.1:35724
I20260810 23:07:08.342293 11376 tablet_service.cc:1514] Processing DeleteTablet for tablet e94e68b6a9c74899bbba3667a1bcbf3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:08 UTC) from {username='slave'} at 127.0.0.1:49946
I20260810 23:07:08.342305 11524 tablet_service.cc:1514] Processing DeleteTablet for tablet e94e68b6a9c74899bbba3667a1bcbf3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:08 UTC) from {username='slave'} at 127.0.0.1:35642
I20260810 23:07:08.342628 11783 tablet_replica.cc:330] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4: stopping tablet replica
I20260810 23:07:08.342729 11783 raft_consensus.cc:2229] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:07:08.342852 11783 raft_consensus.cc:2258] T e94e68b6a9c74899bbba3667a1bcbf3a P 35bc9a2205e94bfa9ef8941daff3c5a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:08.342887 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 11308
I20260810 23:07:08.342549 11785 tablet_replica.cc:330] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: stopping tablet replica
I20260810 23:07:08.347877 11785 raft_consensus.cc:2229] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:08.348012 11785 raft_consensus.cc:2258] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:08.348177 11784 tablet_replica.cc:330] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: stopping tablet replica
I20260810 23:07:08.348304 11784 raft_consensus.cc:2229] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:08.348423 11784 raft_consensus.cc:2258] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:08.348501 11785 ts_tablet_manager.cc:1905] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:08.348832 11784 ts_tablet_manager.cc:1905] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:08.350384 11785 ts_tablet_manager.cc:1918] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:08.350476 11785 log.cc:1196] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/e94e68b6a9c74899bbba3667a1bcbf3a
I20260810 23:07:08.350600 11784 ts_tablet_manager.cc:1918] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:08.350677 11784 log.cc:1196] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/e94e68b6a9c74899bbba3667a1bcbf3a
I20260810 23:07:08.350752 11785 ts_tablet_manager.cc:1939] T e94e68b6a9c74899bbba3667a1bcbf3a P 48889e06937a4c4b96436953be28cbd4: Deleting consensus metadata
I20260810 23:07:08.350945 11784 ts_tablet_manager.cc:1939] T e94e68b6a9c74899bbba3667a1bcbf3a P 2d10e75f7ce44f46a39c3719153077a5: Deleting consensus metadata
I20260810 23:07:08.351220 11224 catalog_manager.cc:4841] TS 48889e06937a4c4b96436953be28cbd4 (127.23.56.67:41179): tablet e94e68b6a9c74899bbba3667a1bcbf3a (table test-table [id=873ca5b7b12a45039c14c6112848ba4e]) successfully deleted
I20260810 23:07:08.351533 11225 catalog_manager.cc:4841] TS 2d10e75f7ce44f46a39c3719153077a5 (127.23.56.66:35909): tablet e94e68b6a9c74899bbba3667a1bcbf3a (table test-table [id=873ca5b7b12a45039c14c6112848ba4e]) successfully deleted
W20260810 23:07:08.352557 11226 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:38209 (error 108)
I20260810 23:07:08.352923 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 11454
I20260810 23:07:08.353075 11751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:08.352918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:38209 (local address 127.0.0.1:49964)
0810 23:07:08.352982 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:08.353035 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:38209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 23:07:08.354624 11226 catalog_manager.cc:4584] TS 35bc9a2205e94bfa9ef8941daff3c5a4 (127.23.56.65:38209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e94e68b6a9c74899bbba3667a1bcbf3a: Network error: Client connection negotiation failed: client connection to 127.23.56.65:38209: connect: Connection refused (error 111)
I20260810 23:07:08.360208 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 11600
I20260810 23:07:08.368526 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 11206
2026-08-10T23:07:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (756 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:08 dist-test-slave-1xrh krb5kdc[11789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:08 dist-test-slave-1xrh krb5kdc[11789](info): set up 2 sockets
Aug 10 23:07:08 dist-test-slave-1xrh krb5kdc[11789](info): commencing operation
krb5kdc: starting...
W20260810 23:07:09.467917 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:09 dist-test-slave-1xrh krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:09.589918 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33703
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33703
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:09.671715 11803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:09.671911 11803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:09.671932 11803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:09.673053 11803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:09.673089 11803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:09.673103 11803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:09.673116 11803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:09.673128 11803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:09.674542 11803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:09.674718 11803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:09.674896 11803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:09.679548 11808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:09.679561 11812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:09.679586 11809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:09.679880 11803 server_base.cc:1034] running on GCE node
I20260810 23:07:09.680138 11803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:09.680435 11803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:09.681604 11803 hybrid_clock.cc:648] HybridClock initialized: now 1786403229681591 us; error 43 us; skew 500 ppm
Aug 10 23:07:09 dist-test-slave-1xrh krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403229, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:09.684186 11803 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:09.684620 11803 webserver.cc:460] Webserver started at http://127.23.56.126:39613/ using document root <none> and password file <none>
I20260810 23:07:09.684876 11803 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:09.684921 11803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:09.685011 11803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:09.685967 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39"
format_stamp: "Formatted at 2026-08-10 23:07:09 on dist-test-slave-1xrh"
server_key: "f5f1df9214b6e06ebe577e8ab7d71ac8"
server_key_iv: "8e8ae217087b4939cfdb11622aa3192a"
server_key_version: "encryptionkey@0"
I20260810 23:07:09.686264 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39"
format_stamp: "Formatted at 2026-08-10 23:07:09 on dist-test-slave-1xrh"
server_key: "f5f1df9214b6e06ebe577e8ab7d71ac8"
server_key_iv: "8e8ae217087b4939cfdb11622aa3192a"
server_key_version: "encryptionkey@0"
I20260810 23:07:09.687587 11803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:07:09.688263 11819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:09.688444 11803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:09.688514 11803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39"
format_stamp: "Formatted at 2026-08-10 23:07:09 on dist-test-slave-1xrh"
server_key: "f5f1df9214b6e06ebe577e8ab7d71ac8"
server_key_iv: "8e8ae217087b4939cfdb11622aa3192a"
server_key_version: "encryptionkey@0"
I20260810 23:07:09.688563 11803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:09.697708 11803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:09.699906 11803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:09.700013 11803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:09.703789 11803 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33703
I20260810 23:07:09.703819 11876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33703 every 8 connection(s)
I20260810 23:07:09.704152 11803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:09.704988 11877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:09.705062 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 11803
I20260810 23:07:09.705194 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:09.705370 23777 external_mini_cluster.cc:1428] Setting key dfdbf5b83e9cca44947d54a09dfd30e2
I20260810 23:07:09.708307 11877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39: Bootstrap starting.
I20260810 23:07:09.709015 11877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:09.709306 11877 log.cc:824] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:09.710042 11877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39: No bootstrap required, opened a new log
Aug 10 23:07:09 dist-test-slave-1xrh krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:09.712320 11877 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33703 } }
I20260810 23:07:09.712493 11877 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:09.712544 11877 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffc5253bdc7e4f15bff9fad3d4f8ec39, State: Initialized, Role: FOLLOWER
I20260810 23:07:09.712680 11877 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33703 } }
I20260810 23:07:09.712769 11877 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:09.712814 11877 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:09.712878 11877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:09.713555 11877 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33703 } }
I20260810 23:07:09.713680 11877 leader_election.cc:302] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffc5253bdc7e4f15bff9fad3d4f8ec39; no voters:
I20260810 23:07:09.713848 11877 leader_election.cc:288] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:09.713881 11883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:09.714011 11883 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [term 1 LEADER]: Becoming Leader. State: Replica: ffc5253bdc7e4f15bff9fad3d4f8ec39, State: Running, Role: LEADER
I20260810 23:07:09.714118 11877 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:09.714095 11883 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33703 } }
I20260810 23:07:09.714454 11883 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffc5253bdc7e4f15bff9fad3d4f8ec39. Latest consensus state: current_term: 1 leader_uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33703 } } }
I20260810 23:07:09.714546 11883 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:09.714468 11884 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffc5253bdc7e4f15bff9fad3d4f8ec39" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33703 } } }
I20260810 23:07:09.714709 11884 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:09.714757 11889 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:09.715334 11889 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:09.716842 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:09.706015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48898 (local address 127.23.56.126:33703)
0810 23:07:09.706196 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:09.706203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:09.706227 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:07:09.706653 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:09.706658 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:09.706701 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:09.706806 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:09.707635 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.708556 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.709620 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.710109 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.710225 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:09.712841 (+ 2616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:09.712846 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:09.712847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:09.712859 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:09.714812 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:09.715243 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:09.715246 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:09.715247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:09.715349 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:09.715616 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:09.715617 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:09.715617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:09.715916 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:09.716064 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:09.716519 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 23:07:09.716653 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":84,"threads_started":1}
I20260810 23:07:09.717159 11889 catalog_manager.cc:1347] Generated new cluster ID: 8efe9a7aab0a4dc880db388d75f7f67d
I20260810 23:07:09.717221 11889 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:09.726547 11889 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:09.727025 11889 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:09.734570 11889 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39: Generated new TSK 0
I20260810 23:07:09.734735 11889 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:07:09.767095 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33703
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:09.868896 11905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:09.869122 11905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:09.869160 11905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:09.870287 11905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:09.870350 11905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:09.870393 11905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:07:09.871865 11905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:09.872135 11905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:09.872363 11905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:09.877168 11913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:09.877175 11910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:09.877229 11911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:09.877334 11905 server_base.cc:1034] running on GCE node
I20260810 23:07:09.877573 11905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:09.877758 11905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:09.878901 11905 hybrid_clock.cc:648] HybridClock initialized: now 1786403229878884 us; error 28 us; skew 500 ppm
Aug 10 23:07:09 dist-test-slave-1xrh krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403229, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:09.881314 11905 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:07:09.881726 11905 webserver.cc:460] Webserver started at http://127.23.56.65:39411/ using document root <none> and password file <none>
I20260810 23:07:09.881959 11905 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:09.882004 11905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:09.882145 11905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:09.883260 11905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "969a1df16e4e46aaa52b46664ea9cbc0"
format_stamp: "Formatted at 2026-08-10 23:07:09 on dist-test-slave-1xrh"
server_key: "6a265639295a21487070bcbaeee70a6c"
server_key_iv: "3a8874654007197af8861ccfab57cf86"
server_key_version: "encryptionkey@0"
I20260810 23:07:09.883607 11905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "969a1df16e4e46aaa52b46664ea9cbc0"
format_stamp: "Formatted at 2026-08-10 23:07:09 on dist-test-slave-1xrh"
server_key: "6a265639295a21487070bcbaeee70a6c"
server_key_iv: "3a8874654007197af8861ccfab57cf86"
server_key_version: "encryptionkey@0"
I20260810 23:07:09.884905 11905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 23:07:09.885612 11921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:09.885792 11905 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:09.885887 11905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "969a1df16e4e46aaa52b46664ea9cbc0"
format_stamp: "Formatted at 2026-08-10 23:07:09 on dist-test-slave-1xrh"
server_key: "6a265639295a21487070bcbaeee70a6c"
server_key_iv: "3a8874654007197af8861ccfab57cf86"
server_key_version: "encryptionkey@0"
I20260810 23:07:09.885960 11905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:09.898182 11905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:09.899922 11905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:09.900077 11905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:09.900291 11905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:09.900585 11905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:09.900620 11905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:09.900641 11905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:09.900686 11905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:09.906512 11905 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:37509
I20260810 23:07:09.906587 12045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:37509 every 8 connection(s)
I20260810 23:07:09.906872 11905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
Aug 10 23:07:09 dist-test-slave-1xrh krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403229, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:09.912886 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 11905
I20260810 23:07:09.912993 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:07:09.913197 23777 external_mini_cluster.cc:1428] Setting key 400c7c1303700b625a5a9690c4cd2046
I20260810 23:07:09.916105 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:09.907584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:42533 (local address 127.23.56.126:33703)
0810 23:07:09.907653 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:09.907655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:09.907845 (+ 190us) server_negotiation.cc:407] Connection header received
0810 23:07:09.908298 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:09.908299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:09.908308 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:09.908363 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:09.909220 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.909714 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.910626 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.910813 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.910862 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:09.912879 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:09.912880 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:09.912881 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:09.912889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:09.914572 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:09.915004 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:09.915006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:09.915006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:09.915071 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:09.915376 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:09.915377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:09.915378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:09.915537 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:09.915609 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:09.915919 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 23:07:09.916010 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:07:09.916232 12048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:09.907541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33703 (local address 127.23.56.65:42533)
0810 23:07:09.907712 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:09.907751 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:09.908214 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:09.908429 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:09.908432 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:09.908964 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:09.909142 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:09.909146 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.909793 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.909794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:09.910551 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:09.910553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:09.910885 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:09.910886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:09.910993 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:09.911436 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:09.911445 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:09.912783 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:09.914683 (+ 1900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:09.914690 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:09.914692 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:09.914926 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:09.915191 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:09.915191 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:09.915192 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:09.915308 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:09.915605 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:09.915610 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:09.915750 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 23:07:09.915871 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 23:07:09.916024 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":100,"threads_started":1}
I20260810 23:07:09.916761 12046 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33703
I20260810 23:07:09.916875 12046 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:09.917070 12046 heartbeater.cc:511] Master 127.23.56.126:33703 requested a full tablet report, sending...
I20260810 23:07:09.917529 11837 ts_manager.cc:194] Registered new tserver with Master: 969a1df16e4e46aaa52b46664ea9cbc0 (127.23.56.65:37509)
I20260810 23:07:09.918112 11837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:42533
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:07:09.938302 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33703
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:10.020489 12051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:10.020720 12051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:10.020761 12051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:10.021917 12051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:10.021986 12051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:10.022030 12051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:07:10.023567 12051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:10.023873 12051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:10.024122 12051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:07:10.029285 12051 server_base.cc:1034] running on GCE node
W20260810 23:07:10.029265 12059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.029408 12057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.029285 12056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.029759 12051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:10.029963 12051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:10.031103 12051 hybrid_clock.cc:648] HybridClock initialized: now 1786403230031075 us; error 44 us; skew 500 ppm
Aug 10 23:07:10 dist-test-slave-1xrh krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403230, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:10.033625 12051 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:07:10.034097 12051 webserver.cc:460] Webserver started at http://127.23.56.66:46827/ using document root <none> and password file <none>
I20260810 23:07:10.034338 12051 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:10.034408 12051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:10.034565 12051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:10.035651 12051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "33abd517927e44aabe198833ff7089b0"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "81fe155d0bdc6f9fb59fa5e9a13772fd"
server_key_iv: "9d42f66ba84310e406237e66e1c22862"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.036037 12051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "33abd517927e44aabe198833ff7089b0"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "81fe155d0bdc6f9fb59fa5e9a13772fd"
server_key_iv: "9d42f66ba84310e406237e66e1c22862"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.037364 12051 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:07:10.038075 12066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.038249 12051 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:10.038353 12051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "33abd517927e44aabe198833ff7089b0"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "81fe155d0bdc6f9fb59fa5e9a13772fd"
server_key_iv: "9d42f66ba84310e406237e66e1c22862"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.038430 12051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:10.048133 12051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:10.049883 12051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:10.050051 12051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:10.050277 12051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:10.050599 12051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:10.050657 12051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.050699 12051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:10.050727 12051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.056681 12051 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:39343
I20260810 23:07:10.056736 12191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:39343 every 8 connection(s)
I20260810 23:07:10.057027 12051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
Aug 10 23:07:10 dist-test-slave-1xrh krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403230, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:10.063205 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 12051
I20260810 23:07:10.063313 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:07:10.063529 23777 external_mini_cluster.cc:1428] Setting key abd43f7721f645b59fb58fc38b1d58d7
I20260810 23:07:10.066656 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.057853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:55307 (local address 127.23.56.126:33703)
0810 23:07:10.057943 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.057945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.058114 (+ 169us) server_negotiation.cc:407] Connection header received
0810 23:07:10.058604 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.058605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.058617 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.058678 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.059590 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.060085 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.061034 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.061241 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.061319 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.063424 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.063426 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.063426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:10.063434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.065185 (+ 1751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:10.065594 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:10.065596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:10.065596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:10.065657 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:10.065897 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:10.065898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:10.065899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:10.066037 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.066107 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.066486 (+ 379us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.066564 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:07:10.066843 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.057798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33703 (local address 127.23.56.66:55307)
0810 23:07:10.057997 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.058030 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.058501 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.058748 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.058751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.059322 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:10.059496 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.059500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.060183 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.060184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.060944 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.060946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.061327 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.061328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.061451 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.061939 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:10.061947 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:10.063337 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:10.065284 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:10.065291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:10.065293 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:10.065519 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:10.065737 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:10.065738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:10.065738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:10.065830 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:10.066149 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:10.066158 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:10.066345 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.066462 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.066615 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
I20260810 23:07:10.067353 12193 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33703
I20260810 23:07:10.067441 12193 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:10.067622 12193 heartbeater.cc:511] Master 127.23.56.126:33703 requested a full tablet report, sending...
I20260810 23:07:10.068035 11837 ts_manager.cc:194] Registered new tserver with Master: 33abd517927e44aabe198833ff7089b0 (127.23.56.66:39343)
I20260810 23:07:10.068567 11837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:55307
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:07:10.090189 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33703
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:10.171582 12199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:10.171805 12199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:10.171846 12199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:10.172986 12199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:10.173050 12199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:10.173096 12199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:07:10.174602 12199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33703
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:10.174851 12199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:10.175083 12199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:10.179916 12205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.179984 12199 server_base.cc:1034] running on GCE node
W20260810 23:07:10.179962 12206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.179916 12208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.180387 12199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:10.180573 12199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:10.181710 12199 hybrid_clock.cc:648] HybridClock initialized: now 1786403230181682 us; error 50 us; skew 500 ppm
Aug 10 23:07:10 dist-test-slave-1xrh krb5kdc[11789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403230, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:10.184176 12199 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:07:10.184620 12199 webserver.cc:460] Webserver started at http://127.23.56.67:40073/ using document root <none> and password file <none>
I20260810 23:07:10.184793 12199 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:10.184832 12199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:10.184926 12199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:10.186064 12199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "a98a14a90d9446f78e7dc21b1cc314da"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "223d67f572539e94425ef863ee5e81ab"
server_key_iv: "a195d544b8dd9c796481afa4b3959c83"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.186395 12199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "a98a14a90d9446f78e7dc21b1cc314da"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "223d67f572539e94425ef863ee5e81ab"
server_key_iv: "a195d544b8dd9c796481afa4b3959c83"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.187827 12199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:10.188623 12216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.188879 12199 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:10.188980 12199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "a98a14a90d9446f78e7dc21b1cc314da"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "223d67f572539e94425ef863ee5e81ab"
server_key_iv: "a195d544b8dd9c796481afa4b3959c83"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.189076 12199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:10.201604 12199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:10.203469 12199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:10.203624 12199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:10.203859 12199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:10.204170 12199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:10.204226 12199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.204267 12199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:10.204296 12199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.210084 12199 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:42237
I20260810 23:07:10.210139 12346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:42237 every 8 connection(s)
I20260810 23:07:10.210469 12199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:07:10.215014 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 12199
I20260810 23:07:10.215121 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:07:10.215348 23777 external_mini_cluster.cc:1428] Setting key 08174ddf5879b4be6874d249c474ab81
Aug 10 23:07:10 dist-test-slave-1xrh krb5kdc[11789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403230, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:10.220129 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.211177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:44075 (local address 127.23.56.126:33703)
0810 23:07:10.211222 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.211224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.211520 (+ 296us) server_negotiation.cc:407] Connection header received
0810 23:07:10.212051 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.212052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.212065 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.212130 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.213095 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.213612 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.214693 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.215171 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.215248 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.217618 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.217619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.217620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:10.217627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.218867 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:10.219321 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:10.219322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:10.219322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:10.219385 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:10.219608 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:10.219608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:10.219609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:10.219718 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.219769 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.220002 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.220067 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:07:10.220286 12349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.211193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33703 (local address 127.23.56.67:44075)
0810 23:07:10.211378 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.211423 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.211948 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.212224 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.212233 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.212801 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:10.212985 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.212990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.213719 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.213720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.214603 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.214606 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.215257 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.215258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.215489 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.216082 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:10.216092 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:10.217532 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:10.218941 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:10.218947 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:10.218949 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:10.219256 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:10.219456 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:10.219457 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:10.219457 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:10.219554 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:10.219772 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:10.219777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:10.219892 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.219974 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.220091 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260810 23:07:10.220768 12347 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33703
I20260810 23:07:10.220880 12347 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:10.221072 12347 heartbeater.cc:511] Master 127.23.56.126:33703 requested a full tablet report, sending...
I20260810 23:07:10.221450 11837 ts_manager.cc:194] Registered new tserver with Master: a98a14a90d9446f78e7dc21b1cc314da (127.23.56.67:42237)
I20260810 23:07:10.221817 11837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:44075
I20260810 23:07:10.227233 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:07:10.236014 11881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.228953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.23.56.126:33703)
0810 23:07:10.229018 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.229021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.229153 (+ 132us) server_negotiation.cc:407] Connection header received
0810 23:07:10.229184 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.229185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.229199 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.229264 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.229797 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.230560 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.231354 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.231663 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.231741 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.233030 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.233031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.233032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:10.233039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.234509 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:10.235055 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:10.235056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:10.235057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:10.235125 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:10.235371 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:10.235373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:10.235373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:10.235566 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.235683 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.235785 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.235862 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:07:10.237792 11837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:07:10.239526 11837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:07:10.245931 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.242069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47588 (local address 127.23.56.67:42237)
0810 23:07:10.242284 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.242289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.242301 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:07:10.242343 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.242344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.242362 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.242455 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:10.242651 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.243487 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.244854 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.245501 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.245588 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.245648 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.245815 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.245854 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":88,"threads_started":1}
I20260810 23:07:10.245932 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.241815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:42237 (local address 127.0.0.1:47588)
0810 23:07:10.242128 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.242139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.242198 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.242474 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.242477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.242480 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:10.242558 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.242560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.243591 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.243592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.244779 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.244782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.245598 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.245600 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.245681 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.245695 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.245796 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.245852 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":10,"thread_start_us":42,"threads_started":1}
I20260810 23:07:10.246205 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.241815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:39343 (local address 127.0.0.1:37954)
0810 23:07:10.241987 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.242027 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.242096 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.242474 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.242477 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.242483 (+ 6us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:10.242558 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.242561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.244029 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.244030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.245035 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.245037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.245998 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.245999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.246059 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.246063 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.246092 (+ 29us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.246134 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260810 23:07:10.246520 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.241828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:37509 (local address 127.0.0.1:52532)
0810 23:07:10.242311 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.242326 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.242363 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.242819 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.242820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.242823 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:10.242882 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.242884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.244141 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.244142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.245144 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.245147 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.246295 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.246296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.246370 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.246375 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.246418 (+ 43us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.246455 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":109,"threads_started":1}
I20260810 23:07:10.246634 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.241939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37954 (local address 127.23.56.66:39343)
0810 23:07:10.242296 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.242300 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.242309 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:07:10.242345 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.242347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.242362 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.242451 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:10.242734 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.243919 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.245123 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.245885 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.246397 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.246481 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.246509 (+ 28us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.246557 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":90,"threads_started":1}
I20260810 23:07:10.246806 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.242117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52532 (local address 127.23.56.65:37509)
0810 23:07:10.242593 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.242598 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.242607 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:07:10.242646 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.242649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.242665 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.242740 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:10.242966 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.244028 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.245229 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.245923 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.245987 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.246651 (+ 664us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.246678 (+ 27us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.246729 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":87,"threads_started":1}
I20260810 23:07:10.246816 12272 tablet_service.cc:1467] Processing CreateTablet for tablet 4c648e7082c44f6795f244146e8ebb96 (DEFAULT_TABLE table=test-table [id=6f7b480701a04d9294aa5a1de8b6e7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:10.247241 12272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c648e7082c44f6795f244146e8ebb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:10.247483 11971 tablet_service.cc:1467] Processing CreateTablet for tablet 4c648e7082c44f6795f244146e8ebb96 (DEFAULT_TABLE table=test-table [id=6f7b480701a04d9294aa5a1de8b6e7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:10.247501 12121 tablet_service.cc:1467] Processing CreateTablet for tablet 4c648e7082c44f6795f244146e8ebb96 (DEFAULT_TABLE table=test-table [id=6f7b480701a04d9294aa5a1de8b6e7b9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:10.247740 11971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c648e7082c44f6795f244146e8ebb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:10.247833 12121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c648e7082c44f6795f244146e8ebb96. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:10.250062 12365 tablet_bootstrap.cc:492] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0: Bootstrap starting.
I20260810 23:07:10.250638 12366 tablet_bootstrap.cc:492] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: Bootstrap starting.
I20260810 23:07:10.250701 12365 tablet_bootstrap.cc:654] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:10.251016 12367 tablet_bootstrap.cc:492] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: Bootstrap starting.
I20260810 23:07:10.251354 12366 tablet_bootstrap.cc:654] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:10.251338 12365 log.cc:824] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:10.251670 12367 tablet_bootstrap.cc:654] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:10.251950 12366 log.cc:824] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:10.252264 12367 log.cc:824] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:10.252677 12366 tablet_bootstrap.cc:492] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: No bootstrap required, opened a new log
I20260810 23:07:10.252767 12366 ts_tablet_manager.cc:1397] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:10.252956 12367 tablet_bootstrap.cc:492] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: No bootstrap required, opened a new log
I20260810 23:07:10.253010 12367 ts_tablet_manager.cc:1397] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:10.254464 12366 raft_consensus.cc:348] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } }
I20260810 23:07:10.254601 12366 raft_consensus.cc:374] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:10.254623 12366 raft_consensus.cc:729] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a98a14a90d9446f78e7dc21b1cc314da, State: Initialized, Role: FOLLOWER
I20260810 23:07:10.254547 12367 raft_consensus.cc:348] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } }
I20260810 23:07:10.254724 12367 raft_consensus.cc:374] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:10.254773 12367 raft_consensus.cc:729] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33abd517927e44aabe198833ff7089b0, State: Initialized, Role: FOLLOWER
I20260810 23:07:10.254742 12366 consensus_queue.cc:260] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } }
I20260810 23:07:10.254894 12367 consensus_queue.cc:260] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } }
I20260810 23:07:10.255033 12366 ts_tablet_manager.cc:1428] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:10.255059 12347 heartbeater.cc:503] Master 127.23.56.126:33703 was elected leader, sending a full tablet report...
I20260810 23:07:10.255204 12367 ts_tablet_manager.cc:1428] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:10.255232 12193 heartbeater.cc:503] Master 127.23.56.126:33703 was elected leader, sending a full tablet report...
I20260810 23:07:10.255617 12365 tablet_bootstrap.cc:492] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0: No bootstrap required, opened a new log
I20260810 23:07:10.255686 12365 ts_tablet_manager.cc:1397] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260810 23:07:10.257364 12365 raft_consensus.cc:348] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } }
I20260810 23:07:10.257547 12365 raft_consensus.cc:374] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:10.257592 12365 raft_consensus.cc:729] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 969a1df16e4e46aaa52b46664ea9cbc0, State: Initialized, Role: FOLLOWER
I20260810 23:07:10.257716 12365 consensus_queue.cc:260] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } }
I20260810 23:07:10.257997 12365 ts_tablet_manager.cc:1428] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:07:10.258040 12046 heartbeater.cc:503] Master 127.23.56.126:33703 was elected leader, sending a full tablet report...
W20260810 23:07:10.307564 12194 tablet.cc:2369] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:07:10.318738 12373 raft_consensus.cc:482] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:07:10.318908 12373 raft_consensus.cc:504] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } }
I20260810 23:07:10.319419 12373 leader_election.cc:288] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a98a14a90d9446f78e7dc21b1cc314da (127.23.56.67:42237), 969a1df16e4e46aaa52b46664ea9cbc0 (127.23.56.65:37509)
I20260810 23:07:10.323364 12364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.319704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:57719 (local address 127.23.56.65:37509)
0810 23:07:10.319753 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.319754 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.319913 (+ 159us) server_negotiation.cc:407] Connection header received
0810 23:07:10.319943 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.319944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.319962 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.320013 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:10.320160 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.320929 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.322381 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.322972 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.323051 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.323060 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.323255 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.323291 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:07:10.323364 12363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.319754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:43943 (local address 127.23.56.67:42237)
0810 23:07:10.319807 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.319809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.319817 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:10.319850 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.319850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.319858 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.319954 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:10.320168 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.320923 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.322376 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.322973 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.323051 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.323060 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.323253 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.323291 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 23:07:10.323410 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.319541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:42237 (local address 127.23.56.66:43943)
0810 23:07:10.319583 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.319596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.319640 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.319985 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.319985 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.319987 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:10.320086 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.320087 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.321016 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.321017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.322280 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.322283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.323065 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.323066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.323184 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.323186 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.323240 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.323290 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":116}
I20260810 23:07:10.323411 12375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.319605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:37509 (local address 127.23.56.66:57719)
0810 23:07:10.319810 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.319825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.319878 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.320033 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.320033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.320034 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:10.320093 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.320094 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.321016 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.321017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.322280 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.322283 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.323065 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.323066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.323179 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.323183 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.323240 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.323290 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":57,"thread_start_us":124,"threads_started":1}
I20260810 23:07:10.323825 12294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c648e7082c44f6795f244146e8ebb96" candidate_uuid: "33abd517927e44aabe198833ff7089b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98a14a90d9446f78e7dc21b1cc314da" is_pre_election: true
I20260810 23:07:10.323997 12294 raft_consensus.cc:2454] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33abd517927e44aabe198833ff7089b0 in term 0.
I20260810 23:07:10.324261 11994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c648e7082c44f6795f244146e8ebb96" candidate_uuid: "33abd517927e44aabe198833ff7089b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" is_pre_election: true
I20260810 23:07:10.324395 11994 raft_consensus.cc:2454] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 33abd517927e44aabe198833ff7089b0 in term 0.
I20260810 23:07:10.324574 12067 leader_election.cc:302] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33abd517927e44aabe198833ff7089b0, a98a14a90d9446f78e7dc21b1cc314da; no voters:
I20260810 23:07:10.324751 12373 raft_consensus.cc:2781] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:07:10.324823 12373 raft_consensus.cc:482] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:07:10.324841 12373 raft_consensus.cc:3037] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:10.325604 12373 raft_consensus.cc:504] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } }
I20260810 23:07:10.325752 12373 leader_election.cc:288] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [CANDIDATE]: Term 1 election: Requested vote from peers a98a14a90d9446f78e7dc21b1cc314da (127.23.56.67:42237), 969a1df16e4e46aaa52b46664ea9cbc0 (127.23.56.65:37509)
I20260810 23:07:10.325948 11994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c648e7082c44f6795f244146e8ebb96" candidate_uuid: "33abd517927e44aabe198833ff7089b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "969a1df16e4e46aaa52b46664ea9cbc0"
I20260810 23:07:10.326054 11994 raft_consensus.cc:3037] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:10.326893 11994 raft_consensus.cc:2454] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33abd517927e44aabe198833ff7089b0 in term 1.
I20260810 23:07:10.327021 12294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c648e7082c44f6795f244146e8ebb96" candidate_uuid: "33abd517927e44aabe198833ff7089b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a98a14a90d9446f78e7dc21b1cc314da"
I20260810 23:07:10.327070 12068 leader_election.cc:302] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33abd517927e44aabe198833ff7089b0, 969a1df16e4e46aaa52b46664ea9cbc0; no voters:
I20260810 23:07:10.327109 12294 raft_consensus.cc:3037] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:10.327193 12373 raft_consensus.cc:2781] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:10.327386 12373 raft_consensus.cc:686] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 1 LEADER]: Becoming Leader. State: Replica: 33abd517927e44aabe198833ff7089b0, State: Running, Role: LEADER
I20260810 23:07:10.327589 12373 consensus_queue.cc:237] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } }
I20260810 23:07:10.328073 12294 raft_consensus.cc:2454] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 33abd517927e44aabe198833ff7089b0 in term 1.
I20260810 23:07:10.328550 11837 catalog_manager.cc:5495] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 33abd517927e44aabe198833ff7089b0 (127.23.56.66). New cstate: current_term: 1 leader_uuid: "33abd517927e44aabe198833ff7089b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33abd517927e44aabe198833ff7089b0" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 } health_report { overall_health: UNKNOWN } } }
I20260810 23:07:10.342031 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.339680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37970 (local address 127.23.56.66:39343)
0810 23:07:10.339728 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.339731 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.339740 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:07:10.339807 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.339808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.339834 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.339918 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:10.340029 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.340492 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.341282 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.341475 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.341552 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.341711 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:10.341802 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:10.341920 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.341950 (+ 30us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.341961 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260810 23:07:10.343863 11994 raft_consensus.cc:1261] T 4c648e7082c44f6795f244146e8ebb96 P 969a1df16e4e46aaa52b46664ea9cbc0 [term 1 FOLLOWER]: Refusing update from remote peer 33abd517927e44aabe198833ff7089b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:10.343907 12294 raft_consensus.cc:1261] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [term 1 FOLLOWER]: Refusing update from remote peer 33abd517927e44aabe198833ff7089b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:10.344240 12376 consensus_queue.cc:1035] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a98a14a90d9446f78e7dc21b1cc314da" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 42237 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:10.344395 12373 consensus_queue.cc:1035] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "969a1df16e4e46aaa52b46664ea9cbc0" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 37509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:10.347363 12381 mvcc.cc:205] Tried to move back new op lower bound from 7317107631485419520 to 7317107631422636032. Current Snapshot: MvccSnapshot[applied={T|T < 7317107631485419520}]
I20260810 23:07:10.349565 11837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:10.349689 11837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:10.350449 11837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffc5253bdc7e4f15bff9fad3d4f8ec39: Sending DeleteTablet for 3 replicas of tablet 4c648e7082c44f6795f244146e8ebb96
I20260810 23:07:10.350835 12121 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c648e7082c44f6795f244146e8ebb96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:10 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:37954
I20260810 23:07:10.350870 11971 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c648e7082c44f6795f244146e8ebb96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:10 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:52532
I20260810 23:07:10.351087 12389 tablet_replica.cc:330] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: stopping tablet replica
I20260810 23:07:10.351198 12389 raft_consensus.cc:2229] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:07:10.351203 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 11905
I20260810 23:07:10.351292 12389 raft_consensus.cc:2258] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:10.353513 12389 ts_tablet_manager.cc:1905] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:10.355531 12389 ts_tablet_manager.cc:1918] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:10.355545 12272 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c648e7082c44f6795f244146e8ebb96 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:10 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:47588
I20260810 23:07:10.355602 12389 log.cc:1196] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/4c648e7082c44f6795f244146e8ebb96
I20260810 23:07:10.355860 12389 ts_tablet_manager.cc:1939] T 4c648e7082c44f6795f244146e8ebb96 P 33abd517927e44aabe198833ff7089b0: Deleting consensus metadata
I20260810 23:07:10.356300 11823 catalog_manager.cc:4841] TS 33abd517927e44aabe198833ff7089b0 (127.23.56.66:39343): tablet 4c648e7082c44f6795f244146e8ebb96 (table test-table [id=6f7b480701a04d9294aa5a1de8b6e7b9]) successfully deleted
W20260810 23:07:10.358816 11821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:37509 (error 108)
I20260810 23:07:10.359181 12390 tablet_replica.cc:330] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: stopping tablet replica
I20260810 23:07:10.359321 12390 raft_consensus.cc:2229] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:10.359480 12390 raft_consensus.cc:2258] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:10.359602 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.359236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:37509 (local address 127.0.0.1:52546)
0810 23:07:10.359509 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.359565 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:37509: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247}
W20260810 23:07:10.359731 11821 catalog_manager.cc:4584] TS 969a1df16e4e46aaa52b46664ea9cbc0 (127.23.56.65:37509): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c648e7082c44f6795f244146e8ebb96: Network error: Client connection negotiation failed: client connection to 127.23.56.65:37509: connect: Connection refused (error 111)
I20260810 23:07:10.359952 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 12051
I20260810 23:07:10.359997 12390 ts_tablet_manager.cc:1905] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:10.361806 12390 ts_tablet_manager.cc:1918] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:10.361917 12390 log.cc:1196] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/4c648e7082c44f6795f244146e8ebb96
I20260810 23:07:10.362174 12390 ts_tablet_manager.cc:1939] T 4c648e7082c44f6795f244146e8ebb96 P a98a14a90d9446f78e7dc21b1cc314da: Deleting consensus metadata
I20260810 23:07:10.362840 11820 catalog_manager.cc:4841] TS a98a14a90d9446f78e7dc21b1cc314da (127.23.56.67:42237): tablet 4c648e7082c44f6795f244146e8ebb96 (table test-table [id=6f7b480701a04d9294aa5a1de8b6e7b9]) successfully deleted
I20260810 23:07:10.368584 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 12199
I20260810 23:07:10.376541 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 11803
2026-08-10T23:07:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2008 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T23:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:10Z Disabled control of system clock
I20260810 23:07:10.407361 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33519
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:07:10.488731 12395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:10.488988 12395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:10.489025 12395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:10.490151 12395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:10.490223 12395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:10.490247 12395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:10.490275 12395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:10.490299 12395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:10.491667 12395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:10.491938 12395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:10.492223 12395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:10.497026 12401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.497035 12400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.497032 12403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.497081 12395 server_base.cc:1034] running on GCE node
I20260810 23:07:10.497422 12395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:10.497716 12395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:10.498893 12395 hybrid_clock.cc:648] HybridClock initialized: now 1786403230498878 us; error 45 us; skew 500 ppm
I20260810 23:07:10.500280 12395 webserver.cc:460] Webserver started at http://127.23.56.126:37975/ using document root <none> and password file <none>
I20260810 23:07:10.500528 12395 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:10.500604 12395 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:10.500747 12395 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:10.502025 12395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "970b79aad91f41cb8b1320f1d5686ab4"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "02b25efdb623f7a74df10991d1c523bd"
server_key_iv: "38a18075c2d388f7183136c52e527c6b"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.502514 12395 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "970b79aad91f41cb8b1320f1d5686ab4"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "02b25efdb623f7a74df10991d1c523bd"
server_key_iv: "38a18075c2d388f7183136c52e527c6b"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.505460 12395 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260810 23:07:10.506379 12410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.506780 12395 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:10.506862 12395 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "970b79aad91f41cb8b1320f1d5686ab4"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "02b25efdb623f7a74df10991d1c523bd"
server_key_iv: "38a18075c2d388f7183136c52e527c6b"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.506986 12395 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:10.524583 12395 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:10.524905 12395 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:10.525058 12395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:10.529083 12395 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33519
I20260810 23:07:10.529146 12467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33519 every 8 connection(s)
I20260810 23:07:10.529491 12395 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:10.530443 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:10.531549 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 12395
I20260810 23:07:10.531626 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:10.531822 23777 external_mini_cluster.cc:1428] Setting key 289874d79c09dd8d67db23bbfbef0997
I20260810 23:07:10.533282 12468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4: Bootstrap starting.
I20260810 23:07:10.533890 12468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:10.534178 12468 log.cc:824] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:10.535527 12468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4: No bootstrap required, opened a new log
I20260810 23:07:10.537348 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.532359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36100 (local address 127.23.56.126:33519)
0810 23:07:10.532578 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.532594 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.532618 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:07:10.533078 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.533084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.533143 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.533294 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.533533 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.534568 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.535491 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.535922 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.536012 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.536700 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.536704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.536704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:10.536858 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.536907 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:10.536917 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.536967 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.537073 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.537166 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":88,"threads_started":1}
I20260810 23:07:10.537883 12468 raft_consensus.cc:348] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970b79aad91f41cb8b1320f1d5686ab4" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33519 } }
I20260810 23:07:10.538036 12468 raft_consensus.cc:374] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:10.538084 12468 raft_consensus.cc:729] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970b79aad91f41cb8b1320f1d5686ab4, State: Initialized, Role: FOLLOWER
I20260810 23:07:10.538236 12468 consensus_queue.cc:260] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970b79aad91f41cb8b1320f1d5686ab4" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33519 } }
I20260810 23:07:10.538327 12468 raft_consensus.cc:388] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:10.538371 12468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:10.538419 12468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:10.538924 12468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970b79aad91f41cb8b1320f1d5686ab4" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33519 } }
I20260810 23:07:10.539050 12468 leader_election.cc:302] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 970b79aad91f41cb8b1320f1d5686ab4; no voters:
I20260810 23:07:10.539258 12468 leader_election.cc:288] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:10.539309 12473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:10.539474 12473 raft_consensus.cc:686] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [term 1 LEADER]: Becoming Leader. State: Replica: 970b79aad91f41cb8b1320f1d5686ab4, State: Running, Role: LEADER
I20260810 23:07:10.539530 12468 sys_catalog.cc:564] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:10.539564 12473 consensus_queue.cc:237] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970b79aad91f41cb8b1320f1d5686ab4" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33519 } }
I20260810 23:07:10.539884 12475 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 970b79aad91f41cb8b1320f1d5686ab4. Latest consensus state: current_term: 1 leader_uuid: "970b79aad91f41cb8b1320f1d5686ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970b79aad91f41cb8b1320f1d5686ab4" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33519 } } }
I20260810 23:07:10.539894 12474 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "970b79aad91f41cb8b1320f1d5686ab4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "970b79aad91f41cb8b1320f1d5686ab4" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33519 } } }
I20260810 23:07:10.539999 12475 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:10.540010 12474 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:10.540302 12482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:10.540813 12482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:10.542323 12482 catalog_manager.cc:1347] Generated new cluster ID: b92330c4e29d4ed0a05f77e4cd380b88
I20260810 23:07:10.542397 12482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:10.563937 12482 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:10.564469 12482 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:10.572855 12482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4: Generated new TSK 0
I20260810 23:07:10.573032 12482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:10.578655 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33519
--builtin_ntp_servers=127.23.56.84:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:07:10.680936 12495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:10.681185 12495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:10.681223 12495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:10.682384 12495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:10.682449 12495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:10.682494 12495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:07:10.683959 12495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:10.684221 12495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:10.684532 12495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:10.690299 12501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.690337 12500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.690321 12503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.690623 12495 server_base.cc:1034] running on GCE node
I20260810 23:07:10.690791 12495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:10.690984 12495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:10.692121 12495 hybrid_clock.cc:648] HybridClock initialized: now 1786403230692110 us; error 26 us; skew 500 ppm
I20260810 23:07:10.698113 12495 webserver.cc:460] Webserver started at http://127.23.56.65:37021/ using document root <none> and password file <none>
I20260810 23:07:10.698355 12495 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:10.698410 12495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:10.698585 12495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:10.699712 12495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "c435c982e5894008a60f8dad207ef346"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "c5ff0511cfda6ecdcdac85c6d1477e53"
server_key_iv: "8828673f082e9703833ecaaac1491465"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.700094 12495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "c435c982e5894008a60f8dad207ef346"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "c5ff0511cfda6ecdcdac85c6d1477e53"
server_key_iv: "8828673f082e9703833ecaaac1491465"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.701387 12495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:07:10.702107 12510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.702287 12495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:10.702399 12495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "c435c982e5894008a60f8dad207ef346"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "c5ff0511cfda6ecdcdac85c6d1477e53"
server_key_iv: "8828673f082e9703833ecaaac1491465"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.702471 12495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:10.716117 12495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:10.716418 12495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:10.716564 12495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:10.716777 12495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:10.717094 12495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:10.717147 12495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.717188 12495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:10.717216 12495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.723256 12495 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:35899
I20260810 23:07:10.723306 12633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:35899 every 8 connection(s)
I20260810 23:07:10.723654 12495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:07:10.725100 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 12495
I20260810 23:07:10.725222 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:07:10.725426 23777 external_mini_cluster.cc:1428] Setting key efd52f3be5f044e7e786afecfb6d5479
I20260810 23:07:10.726449 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33519
--builtin_ntp_servers=127.23.56.84:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:07:10.729317 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.724429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:47943 (local address 127.23.56.126:33519)
0810 23:07:10.724481 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.724484 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.724689 (+ 205us) server_negotiation.cc:407] Connection header received
0810 23:07:10.725230 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.725232 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.725250 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.725306 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.726148 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.726751 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.727755 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.728061 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.728134 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.729003 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.729006 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.729007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:10.729014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.729037 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:10.729043 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.729102 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.729210 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.729238 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260810 23:07:10.729565 12637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.724383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33519 (local address 127.23.56.65:47943)
0810 23:07:10.724563 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.724598 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.725120 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.725392 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.725396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.725413 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:10.726076 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.726083 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.726847 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.726849 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.727664 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.727666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.728155 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.728156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.728275 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.728857 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:10.728857 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:10.728886 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:10.728909 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:10.729109 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:10.729113 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:10.729115 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.729198 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.729300 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
I20260810 23:07:10.730050 12634 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33519
I20260810 23:07:10.730144 12634 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:10.730350 12634 heartbeater.cc:511] Master 127.23.56.126:33519 requested a full tablet report, sending...
I20260810 23:07:10.730923 12429 ts_manager.cc:194] Registered new tserver with Master: c435c982e5894008a60f8dad207ef346 (127.23.56.65:35899)
I20260810 23:07:10.731577 12429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.65:47943
W20260810 23:07:10.812348 12638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:10.812577 12638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:10.812613 12638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:10.813700 12638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:10.813766 12638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:10.813803 12638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:07:10.815171 12638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:10.815454 12638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:10.815748 12638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:10.820770 12647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.820820 12645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.820771 12644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.821038 12638 server_base.cc:1034] running on GCE node
I20260810 23:07:10.821254 12638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:10.821498 12638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:10.822667 12638 hybrid_clock.cc:648] HybridClock initialized: now 1786403230822653 us; error 26 us; skew 500 ppm
I20260810 23:07:10.823943 12638 webserver.cc:460] Webserver started at http://127.23.56.66:41683/ using document root <none> and password file <none>
I20260810 23:07:10.824196 12638 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:10.824265 12638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:10.824425 12638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:10.825455 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "f759ac367cb74008811abf9cc07f80e6"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "971ac5ec7d9e014051f74c21e360266e"
server_key_iv: "f6248d48235a7960a967969ea1aa18a5"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.825831 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "f759ac367cb74008811abf9cc07f80e6"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "971ac5ec7d9e014051f74c21e360266e"
server_key_iv: "f6248d48235a7960a967969ea1aa18a5"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.827087 12638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:10.827840 12653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.828012 12638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:10.828109 12638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "f759ac367cb74008811abf9cc07f80e6"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "971ac5ec7d9e014051f74c21e360266e"
server_key_iv: "f6248d48235a7960a967969ea1aa18a5"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.828186 12638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:10.840723 12638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:10.841032 12638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:10.841189 12638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:10.841431 12638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:10.841755 12638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:10.841809 12638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.841847 12638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:10.841874 12638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.847890 12638 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:39125
I20260810 23:07:10.847929 12776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:39125 every 8 connection(s)
I20260810 23:07:10.848304 12638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:07:10.852682 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 12638
I20260810 23:07:10.852792 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:07:10.853016 23777 external_mini_cluster.cc:1428] Setting key bd30efc657b42b6a7bdd660bc94a0c44
I20260810 23:07:10.853225 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.849024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:48913 (local address 127.23.56.126:33519)
0810 23:07:10.849102 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.849106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.849268 (+ 162us) server_negotiation.cc:407] Connection header received
0810 23:07:10.849705 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.849705 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.849715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.849765 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.850377 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.850867 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.851823 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.852001 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.852050 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.852919 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.852921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.852921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:10.852927 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.852945 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:10.852950 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.853003 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.853104 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.853130 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 23:07:10.853456 12780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.848984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33519 (local address 127.23.56.66:48913)
0810 23:07:10.849158 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.849190 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.849627 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.849813 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.849816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.849824 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:10.850309 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.850313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.850944 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.850945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.851755 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.851757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.852074 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.852075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.852173 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.852750 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:10.852751 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:10.852798 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:10.852828 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:10.853010 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:10.853015 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:10.853017 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.853072 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.853209 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":94,"threads_started":1}
I20260810 23:07:10.853936 12778 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33519
I20260810 23:07:10.854043 12778 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:10.854175 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33519
--builtin_ntp_servers=127.23.56.84:41587
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:07:10.854236 12778 heartbeater.cc:511] Master 127.23.56.126:33519 requested a full tablet report, sending...
I20260810 23:07:10.854591 12429 ts_manager.cc:194] Registered new tserver with Master: f759ac367cb74008811abf9cc07f80e6 (127.23.56.66:39125)
I20260810 23:07:10.855074 12429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.66:48913
W20260810 23:07:10.936336 12781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:10.936532 12781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:10.936553 12781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:10.937688 12781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:10.937731 12781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:10.937783 12781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:07:10.939276 12781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41587
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:33519
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:10.939564 12781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:10.939864 12781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:10.944774 12789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.944785 12787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:10.944788 12786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:10.945145 12781 server_base.cc:1034] running on GCE node
I20260810 23:07:10.945358 12781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:10.945586 12781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:10.946744 12781 hybrid_clock.cc:648] HybridClock initialized: now 1786403230946724 us; error 36 us; skew 500 ppm
I20260810 23:07:10.948110 12781 webserver.cc:460] Webserver started at http://127.23.56.67:36831/ using document root <none> and password file <none>
I20260810 23:07:10.948349 12781 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:10.948443 12781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:10.948589 12781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:10.949636 12781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "08d135fea4a34e2089f13f0fd85105d3"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "81a24cf91907364e726f673388763e70"
server_key_iv: "7a82dba52b7193fc6d8aa6700b9eabc0"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.949990 12781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "08d135fea4a34e2089f13f0fd85105d3"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "81a24cf91907364e726f673388763e70"
server_key_iv: "7a82dba52b7193fc6d8aa6700b9eabc0"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.951947 12781 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260810 23:07:10.952627 12796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.952841 12781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:10.952952 12781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "08d135fea4a34e2089f13f0fd85105d3"
format_stamp: "Formatted at 2026-08-10 23:07:10 on dist-test-slave-1xrh"
server_key: "81a24cf91907364e726f673388763e70"
server_key_iv: "7a82dba52b7193fc6d8aa6700b9eabc0"
server_key_version: "encryptionkey@0"
I20260810 23:07:10.953037 12781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:10.974898 12781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:10.975320 12781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:10.975494 12781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:10.975734 12781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:10.976079 12781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:10.976135 12781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.976176 12781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:10.976205 12781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:10.981863 12781 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:33563
I20260810 23:07:10.981930 12918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:33563 every 8 connection(s)
I20260810 23:07:10.982262 12781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:07:10.987457 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.983072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:45157 (local address 127.23.56.126:33519)
0810 23:07:10.983145 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.983147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.983398 (+ 251us) server_negotiation.cc:407] Connection header received
0810 23:07:10.983881 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.983882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.983893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.983957 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.984641 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.985149 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.986103 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.986316 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.986369 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.987197 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.987199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.987199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:10.987205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.987222 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:10.987227 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.987281 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.987370 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.987395 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:07:10.987650 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.983029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:33519 (local address 127.23.56.67:45157)
0810 23:07:10.983228 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:10.983270 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:10.983761 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:10.984026 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:10.984029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:10.984037 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:10.984536 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.984540 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.985238 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.985239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.986025 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:10.986027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.986397 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.986398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:10.986506 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.987010 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:10.987010 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:10.987038 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:10.987060 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:10.987294 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:10.987299 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:10.987302 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:07:10.987359 (+ 57us) client_negotiation.cc:240] Negotiation successful
0810 23:07:10.987443 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":116,"threads_started":1}
I20260810 23:07:10.988162 12919 heartbeater.cc:347] Connected to a master server at 127.23.56.126:33519
I20260810 23:07:10.988270 12919 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:10.988453 12919 heartbeater.cc:511] Master 127.23.56.126:33519 requested a full tablet report, sending...
I20260810 23:07:10.988843 12429 ts_manager.cc:194] Registered new tserver with Master: 08d135fea4a34e2089f13f0fd85105d3 (127.23.56.67:33563)
I20260810 23:07:10.989215 12429 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.67:45157
I20260810 23:07:10.989549 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 12781
I20260810 23:07:10.989652 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:07:10.989856 23777 external_mini_cluster.cc:1428] Setting key ab8866d3332d1c6458454d19a25c145a
I20260810 23:07:10.990644 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:07:10.994814 12471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:10.992047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.23.56.126:33519)
0810 23:07:10.992107 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:10.992109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:10.992215 (+ 106us) server_negotiation.cc:407] Connection header received
0810 23:07:10.992255 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:10.992256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:10.992264 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:10.992312 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:10.992444 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.992896 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.993604 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:10.993774 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:10.993823 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:10.994569 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:10.994570 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:10.994570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:10.994574 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:10.994586 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:10.994590 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:10.994660 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:10.994747 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:07:10.994757 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 23:07:10.996436 12429 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:07:10.997931 12429 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:07:11.004261 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.000107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:39125 (local address 127.0.0.1:47696)
0810 23:07:11.000284 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.000327 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:11.000398 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:11.000653 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:11.000656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:11.000659 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:11.000764 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.000767 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.001900 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.001901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.003113 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.003116 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.003953 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.003954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.004030 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.004044 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 23:07:11.004120 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 23:07:11.004176 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":61,"threads_started":1}
I20260810 23:07:11.004276 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.000210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47696 (local address 127.23.56.66:39125)
0810 23:07:11.000364 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.000369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.000382 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:07:11.000492 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.000494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.000512 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.000643 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:11.000863 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.001805 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.003217 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.003854 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.003955 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.004055 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.004137 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.004176 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260810 23:07:11.004678 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.000410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50592 (local address 127.23.56.67:33563)
0810 23:07:11.000577 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.000582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.000714 (+ 132us) server_negotiation.cc:407] Connection header received
0810 23:07:11.000756 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.000758 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.000774 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.000845 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:11.001035 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.002201 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.003424 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.004164 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.004240 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.004328 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.004560 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.004598 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":87,"threads_started":1}
I20260810 23:07:11.004688 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.000109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:33563 (local address 127.0.0.1:50592)
0810 23:07:11.000592 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.000602 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:11.000657 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:11.000910 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:11.000911 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:11.000914 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:11.000957 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.000958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.002296 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.002297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.003338 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.003340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.004366 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.004367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.004446 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.004450 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:11.004504 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 23:07:11.004590 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":34,"thread_start_us":111,"threads_started":1}
I20260810 23:07:11.004766 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.000551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.23.56.65:35899)
0810 23:07:11.000927 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.000932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.000942 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:11.000980 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.000982 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.000999 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.001072 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:11.001266 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.002322 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.003524 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.004237 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.004300 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.004374 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.004633 (+ 259us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.004675 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":97,"threads_started":1}
I20260810 23:07:11.004908 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.000315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:35899 (local address 127.0.0.1:32994)
0810 23:07:11.000741 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.000751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:11.000805 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:11.001138 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:11.001138 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:11.001140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:11.001178 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.001180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.002392 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.002393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.003432 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.003435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.004402 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.004403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.004452 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.004455 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:07:11.004504 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 23:07:11.004576 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260810 23:07:11.005283 12703 tablet_service.cc:1467] Processing CreateTablet for tablet 54f3406c962546729313d23c459035e2 (DEFAULT_TABLE table=test-table [id=d927aa422c5242b493b669f2b5ad0648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:11.005604 12560 tablet_service.cc:1467] Processing CreateTablet for tablet 54f3406c962546729313d23c459035e2 (DEFAULT_TABLE table=test-table [id=d927aa422c5242b493b669f2b5ad0648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:11.005721 12703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f3406c962546729313d23c459035e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:11.005869 12560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f3406c962546729313d23c459035e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:11.005775 12846 tablet_service.cc:1467] Processing CreateTablet for tablet 54f3406c962546729313d23c459035e2 (DEFAULT_TABLE table=test-table [id=d927aa422c5242b493b669f2b5ad0648]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:11.006083 12846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 54f3406c962546729313d23c459035e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:11.008725 12935 tablet_bootstrap.cc:492] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346: Bootstrap starting.
I20260810 23:07:11.009167 12937 tablet_bootstrap.cc:492] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: Bootstrap starting.
I20260810 23:07:11.009707 12935 tablet_bootstrap.cc:654] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:11.009795 12937 tablet_bootstrap.cc:654] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:11.010073 12935 log.cc:824] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:11.010118 12937 log.cc:824] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:11.010445 12938 tablet_bootstrap.cc:492] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: Bootstrap starting.
I20260810 23:07:11.010903 12937 tablet_bootstrap.cc:492] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: No bootstrap required, opened a new log
I20260810 23:07:11.010903 12935 tablet_bootstrap.cc:492] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346: No bootstrap required, opened a new log
I20260810 23:07:11.010978 12937 ts_tablet_manager.cc:1397] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:07:11.010978 12935 ts_tablet_manager.cc:1397] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:07:11.011066 12938 tablet_bootstrap.cc:654] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:11.011457 12938 log.cc:824] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:11.012159 12938 tablet_bootstrap.cc:492] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: No bootstrap required, opened a new log
I20260810 23:07:11.012245 12938 ts_tablet_manager.cc:1397] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:07:11.013671 12935 raft_consensus.cc:348] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } }
I20260810 23:07:11.013671 12937 raft_consensus.cc:348] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } }
I20260810 23:07:11.013845 12935 raft_consensus.cc:374] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:11.013845 12937 raft_consensus.cc:374] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:11.013885 12935 raft_consensus.cc:729] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c435c982e5894008a60f8dad207ef346, State: Initialized, Role: FOLLOWER
I20260810 23:07:11.013893 12937 raft_consensus.cc:729] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f759ac367cb74008811abf9cc07f80e6, State: Initialized, Role: FOLLOWER
I20260810 23:07:11.014034 12937 consensus_queue.cc:260] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } }
I20260810 23:07:11.014034 12935 consensus_queue.cc:260] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } }
I20260810 23:07:11.014153 12938 raft_consensus.cc:348] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } }
I20260810 23:07:11.014303 12937 ts_tablet_manager.cc:1428] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:07:11.014303 12935 ts_tablet_manager.cc:1428] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 23:07:11.014317 12938 raft_consensus.cc:374] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:11.014364 12938 raft_consensus.cc:729] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08d135fea4a34e2089f13f0fd85105d3, State: Initialized, Role: FOLLOWER
I20260810 23:07:11.014430 12634 heartbeater.cc:503] Master 127.23.56.126:33519 was elected leader, sending a full tablet report...
I20260810 23:07:11.014483 12778 heartbeater.cc:503] Master 127.23.56.126:33519 was elected leader, sending a full tablet report...
I20260810 23:07:11.014487 12938 consensus_queue.cc:260] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } }
I20260810 23:07:11.014729 12938 ts_tablet_manager.cc:1428] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260810 23:07:11.014846 12919 heartbeater.cc:503] Master 127.23.56.126:33519 was elected leader, sending a full tablet report...
I20260810 23:07:11.086207 12945 raft_consensus.cc:482] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:07:11.086367 12945 raft_consensus.cc:504] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } }
I20260810 23:07:11.086742 12945 leader_election.cc:288] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f759ac367cb74008811abf9cc07f80e6 (127.23.56.66:39125), c435c982e5894008a60f8dad207ef346 (127.23.56.65:35899)
I20260810 23:07:11.090361 12921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.086936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:39125 (local address 127.23.56.67:41331)
0810 23:07:11.086998 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.087011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:11.087057 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:11.087282 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:11.087283 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:11.087285 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:11.087377 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.087379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.088380 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.088381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.089592 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.089595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.090115 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.090116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.090185 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.090188 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:07:11.090235 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:07:11.090269 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":72}
I20260810 23:07:11.090380 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.087024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:41331 (local address 127.23.56.66:39125)
0810 23:07:11.087089 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.087093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.087106 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:07:11.087169 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.087170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.087179 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.087254 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:11.087476 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.088255 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.089682 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.090038 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.090092 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.090098 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.090264 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.090314 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:07:11.090445 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.087190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:50377 (local address 127.23.56.65:35899)
0810 23:07:11.087243 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.087245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.087332 (+ 87us) server_negotiation.cc:407] Connection header received
0810 23:07:11.087395 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.087396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.087409 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.087465 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:11.087630 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.088336 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.089655 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.090051 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.090142 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.090150 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.090333 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.090373 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:07:11.090497 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.087061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:35899 (local address 127.23.56.67:50377)
0810 23:07:11.087257 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.087269 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:11.087326 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:11.087479 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:11.087480 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:11.087481 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:11.087550 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.087551 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.088425 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.088425 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.089569 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:11.089572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.090169 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.090170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:11.090253 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.090258 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:07:11.090314 (+ 56us) client_negotiation.cc:240] Negotiation successful
0810 23:07:11.090349 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":7,"thread_start_us":122,"threads_started":1}
I20260810 23:07:11.090822 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f3406c962546729313d23c459035e2" candidate_uuid: "08d135fea4a34e2089f13f0fd85105d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c435c982e5894008a60f8dad207ef346" is_pre_election: true
I20260810 23:07:11.090910 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f3406c962546729313d23c459035e2" candidate_uuid: "08d135fea4a34e2089f13f0fd85105d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f759ac367cb74008811abf9cc07f80e6" is_pre_election: true
I20260810 23:07:11.090986 12582 raft_consensus.cc:2454] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d135fea4a34e2089f13f0fd85105d3 in term 0.
I20260810 23:07:11.091023 12727 raft_consensus.cc:2454] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08d135fea4a34e2089f13f0fd85105d3 in term 0.
I20260810 23:07:11.091245 12798 leader_election.cc:302] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d135fea4a34e2089f13f0fd85105d3, f759ac367cb74008811abf9cc07f80e6; no voters:
I20260810 23:07:11.091403 12945 raft_consensus.cc:2781] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:07:11.091498 12945 raft_consensus.cc:482] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:07:11.091534 12945 raft_consensus.cc:3037] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:11.092334 12945 raft_consensus.cc:504] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } }
I20260810 23:07:11.092478 12945 leader_election.cc:288] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [CANDIDATE]: Term 1 election: Requested vote from peers f759ac367cb74008811abf9cc07f80e6 (127.23.56.66:39125), c435c982e5894008a60f8dad207ef346 (127.23.56.65:35899)
I20260810 23:07:11.092621 12727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f3406c962546729313d23c459035e2" candidate_uuid: "08d135fea4a34e2089f13f0fd85105d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f759ac367cb74008811abf9cc07f80e6"
I20260810 23:07:11.092651 12582 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "54f3406c962546729313d23c459035e2" candidate_uuid: "08d135fea4a34e2089f13f0fd85105d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c435c982e5894008a60f8dad207ef346"
I20260810 23:07:11.092715 12727 raft_consensus.cc:3037] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:11.092753 12582 raft_consensus.cc:3037] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:11.093431 12727 raft_consensus.cc:2454] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d135fea4a34e2089f13f0fd85105d3 in term 1.
I20260810 23:07:11.093606 12582 raft_consensus.cc:2454] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08d135fea4a34e2089f13f0fd85105d3 in term 1.
I20260810 23:07:11.093638 12798 leader_election.cc:302] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08d135fea4a34e2089f13f0fd85105d3, f759ac367cb74008811abf9cc07f80e6; no voters:
I20260810 23:07:11.093765 12945 raft_consensus.cc:2781] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:11.093936 12945 raft_consensus.cc:686] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 1 LEADER]: Becoming Leader. State: Replica: 08d135fea4a34e2089f13f0fd85105d3, State: Running, Role: LEADER
I20260810 23:07:11.094036 12945 consensus_queue.cc:237] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } }
I20260810 23:07:11.094895 12428 catalog_manager.cc:5495] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 08d135fea4a34e2089f13f0fd85105d3 (127.23.56.67). New cstate: current_term: 1 leader_uuid: "08d135fea4a34e2089f13f0fd85105d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08d135fea4a34e2089f13f0fd85105d3" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 33563 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 } health_report { overall_health: UNKNOWN } } }
W20260810 23:07:11.098788 12779 tablet.cc:2369] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:07:11.120065 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.117628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50608 (local address 127.23.56.67:33563)
0810 23:07:11.117698 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:11.117700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:11.117710 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:11.117783 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:11.117783 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:11.117803 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:11.117882 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:11.118021 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.118487 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.119263 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:11.119455 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:11.119529 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:11.119686 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:11.119778 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:11.119889 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:11.119951 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 23:07:11.119960 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 23:07:11.121863 12582 raft_consensus.cc:1261] T 54f3406c962546729313d23c459035e2 P c435c982e5894008a60f8dad207ef346 [term 1 FOLLOWER]: Refusing update from remote peer 08d135fea4a34e2089f13f0fd85105d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:11.121863 12727 raft_consensus.cc:1261] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [term 1 FOLLOWER]: Refusing update from remote peer 08d135fea4a34e2089f13f0fd85105d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:11.122264 12948 consensus_queue.cc:1035] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c435c982e5894008a60f8dad207ef346" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 35899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:11.122397 12945 consensus_queue.cc:1035] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f759ac367cb74008811abf9cc07f80e6" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 39125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:11.125140 12953 mvcc.cc:205] Tried to move back new op lower bound from 7317107634672394240 to 7317107634561912832. Current Snapshot: MvccSnapshot[applied={T|T < 7317107634672394240}]
I20260810 23:07:11.125412 12952 mvcc.cc:205] Tried to move back new op lower bound from 7317107634672394240 to 7317107634561912832. Current Snapshot: MvccSnapshot[applied={T|T < 7317107634672394240}]
I20260810 23:07:11.126842 12429 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:11.126937 12429 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:11.127697 12429 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 970b79aad91f41cb8b1320f1d5686ab4: Sending DeleteTablet for 3 replicas of tablet 54f3406c962546729313d23c459035e2
I20260810 23:07:11.128080 12703 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f3406c962546729313d23c459035e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:11 UTC) from {username='slave'} at 127.0.0.1:47696
I20260810 23:07:11.128082 12846 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f3406c962546729313d23c459035e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:11 UTC) from {username='slave'} at 127.0.0.1:50592
I20260810 23:07:11.128098 12560 tablet_service.cc:1514] Processing DeleteTablet for tablet 54f3406c962546729313d23c459035e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:11 UTC) from {username='slave'} at 127.0.0.1:32994
I20260810 23:07:11.128391 12963 tablet_replica.cc:330] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: stopping tablet replica
I20260810 23:07:11.128489 12963 raft_consensus.cc:2229] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:07:11.128583 12963 raft_consensus.cc:2258] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:11.128675 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 12495
I20260810 23:07:11.128747 12962 tablet_replica.cc:330] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: stopping tablet replica
I20260810 23:07:11.128837 12962 raft_consensus.cc:2229] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:11.128935 12962 raft_consensus.cc:2258] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:11.129345 12962 ts_tablet_manager.cc:1905] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:11.131543 12962 ts_tablet_manager.cc:1918] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:11.131645 12962 log.cc:1196] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/54f3406c962546729313d23c459035e2
I20260810 23:07:11.131989 12962 ts_tablet_manager.cc:1939] T 54f3406c962546729313d23c459035e2 P f759ac367cb74008811abf9cc07f80e6: Deleting consensus metadata
I20260810 23:07:11.137308 12413 catalog_manager.cc:4841] TS f759ac367cb74008811abf9cc07f80e6 (127.23.56.66:39125): tablet 54f3406c962546729313d23c459035e2 (table test-table [id=d927aa422c5242b493b669f2b5ad0648]) successfully deleted
I20260810 23:07:11.137984 12963 ts_tablet_manager.cc:1905] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:11.139303 12963 ts_tablet_manager.cc:1918] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:11.139364 12963 log.cc:1196] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/54f3406c962546729313d23c459035e2
I20260810 23:07:11.139649 12963 ts_tablet_manager.cc:1939] T 54f3406c962546729313d23c459035e2 P 08d135fea4a34e2089f13f0fd85105d3: Deleting consensus metadata
I20260810 23:07:11.140079 12412 catalog_manager.cc:4841] TS 08d135fea4a34e2089f13f0fd85105d3 (127.23.56.67:33563): tablet 54f3406c962546729313d23c459035e2 (table test-table [id=d927aa422c5242b493b669f2b5ad0648]) successfully deleted
W20260810 23:07:11.143016 12413 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:35899 (error 108)
I20260810 23:07:11.143541 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:11.143415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:35899 (local address 127.0.0.1:33000)
0810 23:07:11.143471 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:11.143514 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:35899: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260810 23:07:11.143658 12413 catalog_manager.cc:4584] TS c435c982e5894008a60f8dad207ef346 (127.23.56.65:35899): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 54f3406c962546729313d23c459035e2: Network error: Client connection negotiation failed: client connection to 127.23.56.65:35899: connect: Connection refused (error 111)
I20260810 23:07:11.147277 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 12638
I20260810 23:07:11.157744 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 12781
I20260810 23:07:11.163538 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 12395
2026-08-10T23:07:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (785 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:11 dist-test-slave-1xrh krb5kdc[12968](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:11 dist-test-slave-1xrh krb5kdc[12968](info): set up 2 sockets
Aug 10 23:07:11 dist-test-slave-1xrh krb5kdc[12968](info): commencing operation
krb5kdc: starting...
W20260810 23:07:12.257421 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:12 dist-test-slave-1xrh krb5kdc[12968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:12.377914 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38653
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:38653
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:12.458902 12982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:12.459092 12982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:12.459115 12982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:12.460193 12982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:12.460227 12982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:12.460242 12982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:12.460253 12982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:12.460264 12982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:12.461642 12982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:38653
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:38653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:12.461818 12982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:12.462025 12982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:12.467361 12987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:12.467490 12988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:12.467540 12990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:12.467628 12982 server_base.cc:1034] running on GCE node
I20260810 23:07:12.467943 12982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:12.468231 12982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:12.469377 12982 hybrid_clock.cc:648] HybridClock initialized: now 1786403232469367 us; error 33 us; skew 500 ppm
Aug 10 23:07:12 dist-test-slave-1xrh krb5kdc[12968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:12.471788 12982 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:12.472203 12982 webserver.cc:460] Webserver started at http://127.23.56.126:36711/ using document root <none> and password file <none>
I20260810 23:07:12.472424 12982 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:12.472467 12982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:12.472620 12982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:12.473639 12982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "d413d9e995d847b6942340bc0b94ffdb"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "d931b39369f55b4607427fd15558527e"
server_key_iv: "e2bd04eeed0e7b0e9f0a934ff134a29e"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.473999 12982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "d413d9e995d847b6942340bc0b94ffdb"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "d931b39369f55b4607427fd15558527e"
server_key_iv: "e2bd04eeed0e7b0e9f0a934ff134a29e"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.475356 12982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 23:07:12.476068 12998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.476239 12982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.476347 12982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "d413d9e995d847b6942340bc0b94ffdb"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "d931b39369f55b4607427fd15558527e"
server_key_iv: "e2bd04eeed0e7b0e9f0a934ff134a29e"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.476420 12982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:12.490809 12982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:12.492727 12982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:12.492887 12982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:12.496774 12982 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:38653
I20260810 23:07:12.496832 13053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:38653 every 8 connection(s)
I20260810 23:07:12.497203 12982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:12.498067 13054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:12.500365 13054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb: Bootstrap starting.
I20260810 23:07:12.500932 13054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:12.501217 13054 log.cc:824] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:12.501897 13054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb: No bootstrap required, opened a new log
I20260810 23:07:12.502584 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 12982
I20260810 23:07:12.502703 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:12.502874 23777 external_mini_cluster.cc:1428] Setting key f31b99b943df716c2d6855fb7f727854
I20260810 23:07:12.504063 13054 raft_consensus.cc:348] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d413d9e995d847b6942340bc0b94ffdb" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38653 } }
I20260810 23:07:12.504233 13054 raft_consensus.cc:374] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:12.504272 13054 raft_consensus.cc:729] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d413d9e995d847b6942340bc0b94ffdb, State: Initialized, Role: FOLLOWER
I20260810 23:07:12.504416 13054 consensus_queue.cc:260] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d413d9e995d847b6942340bc0b94ffdb" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38653 } }
I20260810 23:07:12.504500 13054 raft_consensus.cc:388] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:12.504522 13054 raft_consensus.cc:482] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:12.504588 13054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:12.505661 13054 raft_consensus.cc:504] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d413d9e995d847b6942340bc0b94ffdb" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38653 } }
I20260810 23:07:12.505765 13054 leader_election.cc:302] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d413d9e995d847b6942340bc0b94ffdb; no voters:
I20260810 23:07:12.505908 13054 leader_election.cc:288] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:12.505965 13060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:12.506105 13060 raft_consensus.cc:686] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [term 1 LEADER]: Becoming Leader. State: Replica: d413d9e995d847b6942340bc0b94ffdb, State: Running, Role: LEADER
I20260810 23:07:12.506150 13054 sys_catalog.cc:564] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:12.506217 13060 consensus_queue.cc:237] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d413d9e995d847b6942340bc0b94ffdb" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38653 } }
I20260810 23:07:12.506546 13060 sys_catalog.cc:455] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [sys.catalog]: SysCatalogTable state changed. Reason: New leader d413d9e995d847b6942340bc0b94ffdb. Latest consensus state: current_term: 1 leader_uuid: "d413d9e995d847b6942340bc0b94ffdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d413d9e995d847b6942340bc0b94ffdb" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38653 } } }
I20260810 23:07:12.506619 13060 sys_catalog.cc:458] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:12.506811 13067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:12.507351 13067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:12.507993 13061 sys_catalog.cc:455] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d413d9e995d847b6942340bc0b94ffdb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d413d9e995d847b6942340bc0b94ffdb" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 38653 } } }
I20260810 23:07:12.508598 13061 sys_catalog.cc:458] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:12.509081 13067 catalog_manager.cc:1347] Generated new cluster ID: 38b0470676694dbfa6825aac11774478
I20260810 23:07:12.509145 13067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:07:12 dist-test-slave-1xrh krb5kdc[12968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:12.514109 13067 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:12.514772 13067 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:12.515000 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:12.503413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.23.56.126:38653)
0810 23:07:12.503626 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:12.503637 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:12.503670 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:07:12.504025 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:12.504034 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:12.504097 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:12.504244 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:12.505005 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.506435 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.507654 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.508256 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.508357 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:12.511330 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:12.511334 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:12.511334 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:12.511347 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:12.513351 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:12.513809 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:12.513812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:12.513812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:12.513946 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:12.514170 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:12.514172 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:12.514172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:12.514407 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:12.514604 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:12.514671 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 23:07:12.514810 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260810 23:07:12.526932 13067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb: Generated new TSK 0
I20260810 23:07:12.527158 13067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.56.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
Entry for principal kudu/127.23.56.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab.
I20260810 23:07:12.558416 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38653
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:12.660777 13082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:12.660976 13082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:12.661000 13082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:12.662163 13082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:12.662204 13082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:12.662230 13082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:07:12.663795 13082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.65.keytab
--principal=kudu/127.23.56.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:12.664036 13082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:12.664299 13082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:12.670085 13090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:12.670080 13094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:12.670100 13082 server_base.cc:1034] running on GCE node
W20260810 23:07:12.670325 13089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:12.670586 13082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:12.670820 13082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:12.671978 13082 hybrid_clock.cc:648] HybridClock initialized: now 1786403232671967 us; error 31 us; skew 500 ppm
Aug 10 23:07:12 dist-test-slave-1xrh krb5kdc[12968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:12.674443 13082 init.cc:377] Logged in from keytab as kudu/127.23.56.65@KRBTEST.COM (short username kudu)
I20260810 23:07:12.674851 13082 webserver.cc:460] Webserver started at http://127.23.56.65:37243/ using document root <none> and password file <none>
I20260810 23:07:12.675069 13082 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:12.675154 13082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:12.675293 13082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:12.676312 13082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "4bb29ec50310429d9f6e9bbe8f406e1e"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "ead79cbee3b620edff6c7270595289f2"
server_key_iv: "b8059c10d1ca84d23326fb397faad4df"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.676682 13082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "4bb29ec50310429d9f6e9bbe8f406e1e"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "ead79cbee3b620edff6c7270595289f2"
server_key_iv: "b8059c10d1ca84d23326fb397faad4df"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.677973 13082 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:07:12.678675 13101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.678869 13082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:12.678958 13082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "4bb29ec50310429d9f6e9bbe8f406e1e"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "ead79cbee3b620edff6c7270595289f2"
server_key_iv: "b8059c10d1ca84d23326fb397faad4df"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.679044 13082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:12.699282 13082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:12.701061 13082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:12.701237 13082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:12.701470 13082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:12.701788 13082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:12.701844 13082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.701884 13082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:12.701911 13082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.707952 13082 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:41093
I20260810 23:07:12.708050 13220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:41093 every 8 connection(s)
I20260810 23:07:12.708339 13082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
Aug 10 23:07:12 dist-test-slave-1xrh krb5kdc[12968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.65@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:12.715093 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 13082
I20260810 23:07:12.715497 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:07:12.715719 23777 external_mini_cluster.cc:1428] Setting key c0fdb694c99c0ac7d546585a7378a3d8
I20260810 23:07:12.717584 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:12.709042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:34719 (local address 127.23.56.126:38653)
0810 23:07:12.709117 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:12.709118 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:12.709310 (+ 192us) server_negotiation.cc:407] Connection header received
0810 23:07:12.709807 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:12.709808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:12.709821 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:12.709875 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:12.710718 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.711206 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.712141 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.712328 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.712375 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:12.714361 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:12.714363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:12.714363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:12.714374 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:12.716133 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:12.716582 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:12.716583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:12.716583 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:12.716642 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:12.716924 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:12.716925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:12.716926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:12.717043 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:12.717116 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:12.717396 (+ 280us) server_negotiation.cc:299] Negotiation successful
0810 23:07:12.717496 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:07:12.717741 13223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:12.709001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38653 (local address 127.23.56.65:34719)
0810 23:07:12.709171 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:12.709212 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:12.709724 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:12.709924 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:12.709932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:12.710463 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:12.710639 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:12.710643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.711280 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.711281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:12.712068 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:12.712070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.712395 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.712396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:12.712487 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:12.712895 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:12.712903 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:12.714252 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:12.716220 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:12.716228 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:12.716230 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:12.716502 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:12.716719 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:12.716720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:12.716720 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:12.716831 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:12.717121 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:12.717128 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:12.717258 (+ 130us) client_negotiation.cc:769] Sending connection context
0810 23:07:12.717361 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:07:12.717517 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260810 23:07:12.718436 13221 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38653
I20260810 23:07:12.718540 13221 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:12.718727 13221 heartbeater.cc:511] Master 127.23.56.126:38653 requested a full tablet report, sending...
I20260810 23:07:12.719265 13017 ts_manager.cc:194] Registered new tserver with Master: 4bb29ec50310429d9f6e9bbe8f406e1e (127.23.56.65:41093)
I20260810 23:07:12.719854 13017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.65@KRBTEST.COM'} at 127.23.56.65:34719
WARNING: no policy specified for kudu/127.23.56.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
Entry for principal kudu/127.23.56.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab.
I20260810 23:07:12.742970 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38653
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:12.824788 13226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:12.825021 13226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:12.825062 13226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:12.826201 13226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:12.826267 13226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:12.826311 13226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:07:12.827831 13226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.66.keytab
--principal=kudu/127.23.56.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:12.828109 13226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:12.828337 13226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:12.833236 13231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:12.833242 13236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:12.833281 13232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:12.833390 13226 server_base.cc:1034] running on GCE node
I20260810 23:07:12.833658 13226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:12.833891 13226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:12.835057 13226 hybrid_clock.cc:648] HybridClock initialized: now 1786403232835050 us; error 54 us; skew 500 ppm
Aug 10 23:07:12 dist-test-slave-1xrh krb5kdc[12968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:12.837529 13226 init.cc:377] Logged in from keytab as kudu/127.23.56.66@KRBTEST.COM (short username kudu)
I20260810 23:07:12.837991 13226 webserver.cc:460] Webserver started at http://127.23.56.66:39759/ using document root <none> and password file <none>
I20260810 23:07:12.838229 13226 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:12.838296 13226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:12.838443 13226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:12.839524 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "aa803dc6de2d42cf9b9c1839d208642e"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "bb8a4d81ed2016da4866b6e895402f10"
server_key_iv: "1be06f88511299e370b2f2f6577ce9b7"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.839885 13226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "aa803dc6de2d42cf9b9c1839d208642e"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "bb8a4d81ed2016da4866b6e895402f10"
server_key_iv: "1be06f88511299e370b2f2f6577ce9b7"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.841180 13226 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:12.841904 13243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.842070 13226 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:12.842182 13226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "aa803dc6de2d42cf9b9c1839d208642e"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "bb8a4d81ed2016da4866b6e895402f10"
server_key_iv: "1be06f88511299e370b2f2f6577ce9b7"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.842262 13226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:12.853062 13226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:12.854961 13226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:12.855115 13226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:12.855368 13226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:12.855669 13226 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:12.855722 13226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.855753 13226 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:12.855782 13226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.862003 13226 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:34035
I20260810 23:07:12.862053 13365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:34035 every 8 connection(s)
I20260810 23:07:12.862361 13226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:07:12.867895 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 13226
I20260810 23:07:12.868006 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:07:12.868175 23777 external_mini_cluster.cc:1428] Setting key 91a067abc70a3cf0624c9cc2bf6a053a
Aug 10 23:07:12 dist-test-slave-1xrh krb5kdc[12968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.66@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:12.871783 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:12.863004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:48809 (local address 127.23.56.126:38653)
0810 23:07:12.863094 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:12.863097 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:12.863335 (+ 238us) server_negotiation.cc:407] Connection header received
0810 23:07:12.863899 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:12.863900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:12.863914 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:12.863970 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:12.864887 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.865408 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.866388 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.866602 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.866691 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:12.869141 (+ 2450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:12.869142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:12.869143 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:12.869150 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:12.870378 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:12.870790 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:12.870791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:12.870792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:12.870855 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:12.871115 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:12.871116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:12.871116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:12.871256 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:12.871331 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:12.871608 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 23:07:12.871696 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:07:12.871955 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:12.862968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38653 (local address 127.23.56.66:48809)
0810 23:07:12.863177 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:12.863227 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:12.863733 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:12.864036 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:12.864045 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:12.864602 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:12.864782 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:12.864787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.865508 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.865509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:12.866289 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:12.866292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:12.866704 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:12.866705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:12.866835 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:12.867338 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:12.867346 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:12.869043 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:12.870485 (+ 1442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:12.870492 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:12.870493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:12.870710 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:12.870942 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:12.870942 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:12.870943 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:12.871048 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:12.871334 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:12.871341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:12.871473 (+ 132us) client_negotiation.cc:769] Sending connection context
0810 23:07:12.871574 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 23:07:12.871722 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260810 23:07:12.872401 13366 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38653
I20260810 23:07:12.872486 13366 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:12.872669 13366 heartbeater.cc:511] Master 127.23.56.126:38653 requested a full tablet report, sending...
I20260810 23:07:12.873082 13017 ts_manager.cc:194] Registered new tserver with Master: aa803dc6de2d42cf9b9c1839d208642e (127.23.56.66:34035)
I20260810 23:07:12.873459 13017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.66@KRBTEST.COM'} at 127.23.56.66:48809
WARNING: no policy specified for kudu/127.23.56.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
Entry for principal kudu/127.23.56.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab.
I20260810 23:07:12.894102 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38653
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:40831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:12.975574 13372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:12.975759 13372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:12.975782 13372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:12.976879 13372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:12.976920 13372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:12.976946 13372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:07:12.978407 13372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:40831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.67.keytab
--principal=kudu/127.23.56.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:38653
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:12.978587 13372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:12.978832 13372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:12.983817 13377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:12.983840 13381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:12.983824 13378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:12.984165 13372 server_base.cc:1034] running on GCE node
I20260810 23:07:12.984380 13372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:12.984627 13372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:12.985783 13372 hybrid_clock.cc:648] HybridClock initialized: now 1786403232985763 us; error 32 us; skew 500 ppm
Aug 10 23:07:12 dist-test-slave-1xrh krb5kdc[12968](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:12.988382 13372 init.cc:377] Logged in from keytab as kudu/127.23.56.67@KRBTEST.COM (short username kudu)
I20260810 23:07:12.988904 13372 webserver.cc:460] Webserver started at http://127.23.56.67:32817/ using document root <none> and password file <none>
I20260810 23:07:12.989158 13372 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:12.989226 13372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:12.989383 13372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:12.990422 13372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "02f6d013f0dc47e78284b3739e6f13bd"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "a35d048ad1e3832b32e2fd155f594958"
server_key_iv: "1608e9102c5037694624b69180dd7275"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.990794 13372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "02f6d013f0dc47e78284b3739e6f13bd"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "a35d048ad1e3832b32e2fd155f594958"
server_key_iv: "1608e9102c5037694624b69180dd7275"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.992213 13372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:07:12.992950 13390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:12.993121 13372 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:12.993216 13372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "02f6d013f0dc47e78284b3739e6f13bd"
format_stamp: "Formatted at 2026-08-10 23:07:12 on dist-test-slave-1xrh"
server_key: "a35d048ad1e3832b32e2fd155f594958"
server_key_iv: "1608e9102c5037694624b69180dd7275"
server_key_version: "encryptionkey@0"
I20260810 23:07:12.993304 13372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:13.014640 13372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:13.016700 13372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:13.016876 13372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:13.017120 13372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:13.017449 13372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:13.017506 13372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.017551 13372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:13.017575 13372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.023520 13372 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:40381
I20260810 23:07:13.023568 13510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:40381 every 8 connection(s)
I20260810 23:07:13.024398 13372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:07:13.028852 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 13372
I20260810 23:07:13.028952 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:07:13.029105 23777 external_mini_cluster.cc:1428] Setting key 89772ea0fbc9a90118c8d73f75736372
Aug 10 23:07:13 dist-test-slave-1xrh krb5kdc[12968](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403232, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.67@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:13.033674 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.025100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:48259 (local address 127.23.56.126:38653)
0810 23:07:13.025144 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.025146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.025330 (+ 184us) server_negotiation.cc:407] Connection header received
0810 23:07:13.025878 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.025879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.025889 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.025948 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:13.026884 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.027467 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.028376 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.028552 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.028603 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.031027 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:13.031028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:13.031029 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:13.031036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:13.032274 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:13.032665 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:13.032666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:13.032667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:13.032723 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:13.033041 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:13.033042 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:13.033042 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:13.033159 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:13.033234 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.033498 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.033586 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:07:13.033847 13513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.025059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:38653 (local address 127.23.56.67:48259)
0810 23:07:13.025219 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.025260 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.025752 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.026025 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.026028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.026589 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:13.026789 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.026794 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.027538 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.027539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.028305 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.028307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.028625 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.028625 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.028722 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.029217 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:13.029227 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:13.030919 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:13.032358 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:13.032365 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:13.032367 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:13.032584 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:13.032857 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:13.032858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:13.032858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:13.032971 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:13.033226 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:13.033232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:13.033370 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.033475 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.033623 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260810 23:07:13.034363 13511 heartbeater.cc:347] Connected to a master server at 127.23.56.126:38653
I20260810 23:07:13.034469 13511 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:13.034647 13511 heartbeater.cc:511] Master 127.23.56.126:38653 requested a full tablet report, sending...
I20260810 23:07:13.035028 13017 ts_manager.cc:194] Registered new tserver with Master: 02f6d013f0dc47e78284b3739e6f13bd (127.23.56.67:40381)
I20260810 23:07:13.035398 13017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.56.67@KRBTEST.COM'} at 127.23.56.67:48259
I20260810 23:07:13.040732 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:07:13.047653 13059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.042044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54166 (local address 127.23.56.126:38653)
0810 23:07:13.042099 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.042100 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.042185 (+ 85us) server_negotiation.cc:407] Connection header received
0810 23:07:13.042233 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.042233 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.042241 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.042277 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:13.042830 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.043346 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.044102 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.044271 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.044317 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.045416 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:13.045417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:13.045418 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:13.045424 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:13.046688 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:13.046982 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:13.046983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:13.046983 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:13.047029 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:13.047245 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:13.047245 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:13.047245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:13.047371 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:13.047448 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.047540 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.047595 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 23:07:13.049264 13017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:07:13.050853 13017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:07:13.057303 13524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.053045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:40381 (local address 127.0.0.1:45030)
0810 23:07:13.053792 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.053803 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.053850 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.054070 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.054071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.054073 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.054120 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.054122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.055227 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.055228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.056105 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.056108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.057020 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.057021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.057075 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.057086 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.057174 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.057228 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":12,"thread_start_us":115,"threads_started":1}
I20260810 23:07:13.057319 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.053284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45030 (local address 127.23.56.67:40381)
0810 23:07:13.053463 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.053466 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.053944 (+ 478us) server_negotiation.cc:407] Connection header received
0810 23:07:13.053983 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.053986 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.054000 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.054069 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.054186 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.055108 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.056185 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.056938 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.057015 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.057073 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.057204 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.057246 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260810 23:07:13.057799 13526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.053238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39026 (local address 127.23.56.66:34035)
0810 23:07:13.053423 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.053440 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.053549 (+ 109us) server_negotiation.cc:407] Connection header received
0810 23:07:13.053619 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.053621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.053636 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.053727 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.054153 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.054815 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.056337 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.057066 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.057132 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.057189 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.057349 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.057726 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":79,"threads_started":1}
I20260810 23:07:13.057964 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.053045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:34035 (local address 127.0.0.1:39026)
0810 23:07:13.053480 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.053488 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.053553 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.053877 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.053878 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.053880 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.053925 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.053927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.054915 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.054917 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.056247 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.056251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.057146 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.057146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.057202 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.057206 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.057254 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.057285 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":67,"thread_start_us":93,"threads_started":1}
I20260810 23:07:13.058117 13440 tablet_service.cc:1467] Processing CreateTablet for tablet 973f5e84a0f64ed4ae73e412b34f9e22 (DEFAULT_TABLE table=test-table [id=553c5270cd354780b35478cfd9995133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:13.058373 13523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.053045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:41093 (local address 127.0.0.1:54108)
0810 23:07:13.053216 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.053253 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.053322 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.053638 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.053641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.053644 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.053705 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.053707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.054962 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.054963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.056248 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.056251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.058117 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.058118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.058195 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.058200 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.058248 (+ 48us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.058292 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":23,"thread_start_us":54,"threads_started":1}
I20260810 23:07:13.058459 13440 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973f5e84a0f64ed4ae73e412b34f9e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:13.058642 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.053252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54108 (local address 127.23.56.65:41093)
0810 23:07:13.053458 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.053463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.053474 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:07:13.053513 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.053515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.053529 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.053632 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.053790 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.054875 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.057225 (+ 2350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.057957 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.058022 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.058080 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.058515 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.058561 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":88,"threads_started":1}
I20260810 23:07:13.058581 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 973f5e84a0f64ed4ae73e412b34f9e22 (DEFAULT_TABLE table=test-table [id=553c5270cd354780b35478cfd9995133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:13.058874 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973f5e84a0f64ed4ae73e412b34f9e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:13.059216 13150 tablet_service.cc:1467] Processing CreateTablet for tablet 973f5e84a0f64ed4ae73e412b34f9e22 (DEFAULT_TABLE table=test-table [id=553c5270cd354780b35478cfd9995133]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:13.059459 13150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 973f5e84a0f64ed4ae73e412b34f9e22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:13.061822 13530 tablet_bootstrap.cc:492] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: Bootstrap starting.
I20260810 23:07:13.062124 13531 tablet_bootstrap.cc:492] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e: Bootstrap starting.
I20260810 23:07:13.062238 13532 tablet_bootstrap.cc:492] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: Bootstrap starting.
I20260810 23:07:13.062505 13530 tablet_bootstrap.cc:654] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:13.062813 13531 tablet_bootstrap.cc:654] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:13.062817 13532 tablet_bootstrap.cc:654] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:13.063062 13530 log.cc:824] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:13.063450 13532 log.cc:824] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:13.063445 13531 log.cc:824] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:13.063933 13530 tablet_bootstrap.cc:492] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: No bootstrap required, opened a new log
I20260810 23:07:13.064008 13530 ts_tablet_manager.cc:1397] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:13.064231 13532 tablet_bootstrap.cc:492] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: No bootstrap required, opened a new log
I20260810 23:07:13.064303 13532 ts_tablet_manager.cc:1397] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:07:13.064494 13531 tablet_bootstrap.cc:492] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e: No bootstrap required, opened a new log
I20260810 23:07:13.064599 13531 ts_tablet_manager.cc:1397] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 23:07:13.065553 13530 raft_consensus.cc:348] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } }
I20260810 23:07:13.065706 13530 raft_consensus.cc:374] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:13.065749 13530 raft_consensus.cc:729] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02f6d013f0dc47e78284b3739e6f13bd, State: Initialized, Role: FOLLOWER
I20260810 23:07:13.065866 13530 consensus_queue.cc:260] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } }
I20260810 23:07:13.066138 13530 ts_tablet_manager.cc:1428] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:13.066274 13511 heartbeater.cc:503] Master 127.23.56.126:38653 was elected leader, sending a full tablet report...
I20260810 23:07:13.066234 13532 raft_consensus.cc:348] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } }
I20260810 23:07:13.066419 13532 raft_consensus.cc:374] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:13.066479 13532 raft_consensus.cc:729] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa803dc6de2d42cf9b9c1839d208642e, State: Initialized, Role: FOLLOWER
I20260810 23:07:13.066596 13532 consensus_queue.cc:260] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } }
I20260810 23:07:13.066519 13531 raft_consensus.cc:348] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } }
I20260810 23:07:13.066679 13531 raft_consensus.cc:374] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:13.066725 13531 raft_consensus.cc:729] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb29ec50310429d9f6e9bbe8f406e1e, State: Initialized, Role: FOLLOWER
I20260810 23:07:13.066867 13532 ts_tablet_manager.cc:1428] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:07:13.066866 13531 consensus_queue.cc:260] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } }
I20260810 23:07:13.066908 13366 heartbeater.cc:503] Master 127.23.56.126:38653 was elected leader, sending a full tablet report...
I20260810 23:07:13.067137 13531 ts_tablet_manager.cc:1428] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 23:07:13.067253 13221 heartbeater.cc:503] Master 127.23.56.126:38653 was elected leader, sending a full tablet report...
W20260810 23:07:13.113057 13367 tablet.cc:2369] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 23:07:13.209077 13222 tablet.cc:2369] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:07:13.212597 13536 raft_consensus.cc:482] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:07:13.212713 13536 raft_consensus.cc:504] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } }
I20260810 23:07:13.213126 13536 leader_election.cc:288] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa803dc6de2d42cf9b9c1839d208642e (127.23.56.66:34035), 4bb29ec50310429d9f6e9bbe8f406e1e (127.23.56.65:41093)
I20260810 23:07:13.216120 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.213539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:57315 (local address 127.23.56.65:41093)
0810 23:07:13.213618 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.213620 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.213633 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:07:13.213659 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.213660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.213668 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.213732 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.213915 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.214401 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.215369 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.215744 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.215828 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.215838 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.216009 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.216051 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:07:13.216120 13513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.213364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:41093 (local address 127.23.56.67:57315)
0810 23:07:13.213450 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.213465 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.213529 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.213737 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.213738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.213740 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.213820 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.213827 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.214485 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.214486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.215266 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.215268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.215826 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.215828 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.215926 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.215930 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.216000 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.216050 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 23:07:13.216410 13526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.213462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:45233 (local address 127.23.56.66:34035)
0810 23:07:13.213523 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.213526 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.213814 (+ 288us) server_negotiation.cc:407] Connection header received
0810 23:07:13.213889 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.213890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.213899 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.213969 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.214125 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.214620 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.215628 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.216015 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.216107 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.216116 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.216283 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.216329 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260810 23:07:13.216409 13539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.213372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:34035 (local address 127.23.56.67:45233)
0810 23:07:13.213731 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.213740 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.213820 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.213974 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.213975 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.213977 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.214052 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.214053 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.214712 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.214713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.215536 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.215539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.216108 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.216109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.216205 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.216209 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.216283 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.216332 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":169,"thread_start_us":124,"threads_started":1}
I20260810 23:07:13.216504 13171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973f5e84a0f64ed4ae73e412b34f9e22" candidate_uuid: "02f6d013f0dc47e78284b3739e6f13bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" is_pre_election: true
I20260810 23:07:13.216673 13171 raft_consensus.cc:2454] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f6d013f0dc47e78284b3739e6f13bd in term 0.
I20260810 23:07:13.216809 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973f5e84a0f64ed4ae73e412b34f9e22" candidate_uuid: "02f6d013f0dc47e78284b3739e6f13bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa803dc6de2d42cf9b9c1839d208642e" is_pre_election: true
I20260810 23:07:13.216987 13316 raft_consensus.cc:2454] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02f6d013f0dc47e78284b3739e6f13bd in term 0.
I20260810 23:07:13.217067 13391 leader_election.cc:302] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f6d013f0dc47e78284b3739e6f13bd, 4bb29ec50310429d9f6e9bbe8f406e1e; no voters:
I20260810 23:07:13.217216 13536 raft_consensus.cc:2781] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:07:13.217303 13536 raft_consensus.cc:482] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:07:13.217324 13536 raft_consensus.cc:3037] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:13.218119 13536 raft_consensus.cc:504] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } }
I20260810 23:07:13.218271 13536 leader_election.cc:288] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [CANDIDATE]: Term 1 election: Requested vote from peers aa803dc6de2d42cf9b9c1839d208642e (127.23.56.66:34035), 4bb29ec50310429d9f6e9bbe8f406e1e (127.23.56.65:41093)
I20260810 23:07:13.218451 13171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973f5e84a0f64ed4ae73e412b34f9e22" candidate_uuid: "02f6d013f0dc47e78284b3739e6f13bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e"
I20260810 23:07:13.218458 13316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "973f5e84a0f64ed4ae73e412b34f9e22" candidate_uuid: "02f6d013f0dc47e78284b3739e6f13bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa803dc6de2d42cf9b9c1839d208642e"
I20260810 23:07:13.218554 13171 raft_consensus.cc:3037] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:13.218551 13316 raft_consensus.cc:3037] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:13.219333 13171 raft_consensus.cc:2454] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f6d013f0dc47e78284b3739e6f13bd in term 1.
I20260810 23:07:13.219376 13316 raft_consensus.cc:2454] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02f6d013f0dc47e78284b3739e6f13bd in term 1.
I20260810 23:07:13.219534 13391 leader_election.cc:302] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02f6d013f0dc47e78284b3739e6f13bd, 4bb29ec50310429d9f6e9bbe8f406e1e; no voters:
I20260810 23:07:13.219643 13536 raft_consensus.cc:2781] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:13.219805 13536 raft_consensus.cc:686] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 1 LEADER]: Becoming Leader. State: Replica: 02f6d013f0dc47e78284b3739e6f13bd, State: Running, Role: LEADER
I20260810 23:07:13.219879 13536 consensus_queue.cc:237] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } }
I20260810 23:07:13.220767 13016 catalog_manager.cc:5495] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd reported cstate change: term changed from 0 to 1, leader changed from <none> to 02f6d013f0dc47e78284b3739e6f13bd (127.23.56.67). New cstate: current_term: 1 leader_uuid: "02f6d013f0dc47e78284b3739e6f13bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02f6d013f0dc47e78284b3739e6f13bd" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 40381 } health_report { overall_health: HEALTHY } } }
I20260810 23:07:13.236374 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.234192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45040 (local address 127.23.56.67:40381)
0810 23:07:13.234236 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.234237 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.234245 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:13.234312 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.234312 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.234329 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.234386 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:13.234495 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.234955 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.235704 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.235884 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.235947 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.236064 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:13.236150 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:13.236205 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.236234 (+ 29us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.236250 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:07:13.238117 13316 raft_consensus.cc:1261] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [term 1 FOLLOWER]: Refusing update from remote peer 02f6d013f0dc47e78284b3739e6f13bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:13.238134 13171 raft_consensus.cc:1261] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [term 1 FOLLOWER]: Refusing update from remote peer 02f6d013f0dc47e78284b3739e6f13bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:13.238516 13536 consensus_queue.cc:1035] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa803dc6de2d42cf9b9c1839d208642e" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 34035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:13.238662 13540 consensus_queue.cc:1035] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bb29ec50310429d9f6e9bbe8f406e1e" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 41093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:13.241472 13543 mvcc.cc:205] Tried to move back new op lower bound from 7317107643340664832 to 7317107643269324800. Current Snapshot: MvccSnapshot[applied={T|T < 7317107643340664832 or (T in {7317107643340664832})}]
I20260810 23:07:13.243253 13016 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:13.243347 13016 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54166:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:13.244087 13016 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d413d9e995d847b6942340bc0b94ffdb: Sending DeleteTablet for 3 replicas of tablet 973f5e84a0f64ed4ae73e412b34f9e22
I20260810 23:07:13.244467 13150 tablet_service.cc:1514] Processing DeleteTablet for tablet 973f5e84a0f64ed4ae73e412b34f9e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:13 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:54108
I20260810 23:07:13.244453 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet 973f5e84a0f64ed4ae73e412b34f9e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:13 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:39026
I20260810 23:07:13.244705 13553 tablet_replica.cc:330] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e: stopping tablet replica
I20260810 23:07:13.244726 13440 tablet_service.cc:1514] Processing DeleteTablet for tablet 973f5e84a0f64ed4ae73e412b34f9e22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:13 UTC) from {username='kudu', principal='kudu/127.23.56.126@KRBTEST.COM'} at 127.0.0.1:45030
I20260810 23:07:13.244849 13553 raft_consensus.cc:2229] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:13.244987 13553 raft_consensus.cc:2258] T 973f5e84a0f64ed4ae73e412b34f9e22 P 4bb29ec50310429d9f6e9bbe8f406e1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:13.245025 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 13082
I20260810 23:07:13.245177 13554 tablet_replica.cc:330] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: stopping tablet replica
I20260810 23:07:13.245321 13554 raft_consensus.cc:2229] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:13.245436 13554 raft_consensus.cc:2258] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:13.248857 13555 tablet_replica.cc:330] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: stopping tablet replica
I20260810 23:07:13.249017 13555 raft_consensus.cc:2229] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 1 LEADER]: Raft consensus shutting down.
I20260810 23:07:13.249159 13555 raft_consensus.cc:2258] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:13.249161 13554 ts_tablet_manager.cc:1905] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:13.249766 13555 ts_tablet_manager.cc:1905] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:13.250985 13554 ts_tablet_manager.cc:1918] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:13.251061 13554 log.cc:1196] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/973f5e84a0f64ed4ae73e412b34f9e22
I20260810 23:07:13.251438 13555 ts_tablet_manager.cc:1918] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:13.251483 13554 ts_tablet_manager.cc:1939] T 973f5e84a0f64ed4ae73e412b34f9e22 P aa803dc6de2d42cf9b9c1839d208642e: Deleting consensus metadata
I20260810 23:07:13.251514 13555 log.cc:1196] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/973f5e84a0f64ed4ae73e412b34f9e22
I20260810 23:07:13.251781 13555 ts_tablet_manager.cc:1939] T 973f5e84a0f64ed4ae73e412b34f9e22 P 02f6d013f0dc47e78284b3739e6f13bd: Deleting consensus metadata
I20260810 23:07:13.251921 13002 catalog_manager.cc:4841] TS aa803dc6de2d42cf9b9c1839d208642e (127.23.56.66:34035): tablet 973f5e84a0f64ed4ae73e412b34f9e22 (table test-table [id=553c5270cd354780b35478cfd9995133]) successfully deleted
I20260810 23:07:13.252153 13001 catalog_manager.cc:4841] TS 02f6d013f0dc47e78284b3739e6f13bd (127.23.56.67:40381): tablet 973f5e84a0f64ed4ae73e412b34f9e22 (table test-table [id=553c5270cd354780b35478cfd9995133]) successfully deleted
W20260810 23:07:13.255290 13000 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:41093 (error 108)
I20260810 23:07:13.255585 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 13226
I20260810 23:07:13.257833 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.255691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:41093 (local address 127.0.0.1:54118)
0810 23:07:13.257738 (+ 2047us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.257793 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:41093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2021}
W20260810 23:07:13.259176 13000 catalog_manager.cc:4584] TS 4bb29ec50310429d9f6e9bbe8f406e1e (127.23.56.65:41093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 973f5e84a0f64ed4ae73e412b34f9e22: Network error: Client connection negotiation failed: client connection to 127.23.56.65:41093: connect: Connection refused (error 111)
I20260810 23:07:13.270143 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 13372
I20260810 23:07:13.275831 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 12982
2026-08-10T23:07:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2112 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T23:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:13Z Disabled control of system clock
I20260810 23:07:13.305619 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:46205
--webserver_interface=127.23.56.126
--webserver_port=0
--builtin_ntp_servers=127.23.56.84:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:46205
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:07:13.389947 13560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:13.390159 13560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:13.390180 13560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:13.391310 13560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:13.391348 13560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:13.391361 13560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:13.391373 13560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:13.391386 13560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:13.392717 13560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:46205
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:46205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:13.392944 13560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:13.393187 13560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:13.397927 13567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.397997 13560 server_base.cc:1034] running on GCE node
W20260810 23:07:13.397964 13566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.397969 13569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.398401 13560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:13.398708 13560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:13.399856 13560 hybrid_clock.cc:648] HybridClock initialized: now 1786403233399839 us; error 48 us; skew 500 ppm
I20260810 23:07:13.401032 13560 webserver.cc:460] Webserver started at http://127.23.56.126:36893/ using document root <none> and password file <none>
I20260810 23:07:13.401281 13560 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:13.401350 13560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:13.401506 13560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:13.402519 13560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "c81d0f633d50437b9b8c4b2dfa4b4444"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "807b93e3cbfcaa6f5d1223cbda20222e"
server_key_iv: "6682360e3c964030d09516569e0eb2e0"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.402876 13560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "c81d0f633d50437b9b8c4b2dfa4b4444"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "807b93e3cbfcaa6f5d1223cbda20222e"
server_key_iv: "6682360e3c964030d09516569e0eb2e0"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.404237 13560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:07:13.404974 13575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.405172 13560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:13.405270 13560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "c81d0f633d50437b9b8c4b2dfa4b4444"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "807b93e3cbfcaa6f5d1223cbda20222e"
server_key_iv: "6682360e3c964030d09516569e0eb2e0"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.405356 13560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:13.424253 13560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:13.424566 13560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:13.424718 13560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:13.428709 13560 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:46205
I20260810 23:07:13.428743 13633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:46205 every 8 connection(s)
I20260810 23:07:13.429088 13560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:13.429944 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 13560
I20260810 23:07:13.430066 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:13.430083 13634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:13.430258 23777 external_mini_cluster.cc:1428] Setting key aa51b9c9e1d68045773809e1f00a0804
I20260810 23:07:13.433194 13634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444: Bootstrap starting.
I20260810 23:07:13.433854 13634 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:13.434194 13634 log.cc:824] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:13.434947 13634 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444: No bootstrap required, opened a new log
I20260810 23:07:13.435956 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.430847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55658 (local address 127.23.56.126:46205)
0810 23:07:13.431033 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.431047 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.431072 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:07:13.431567 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.431574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.431633 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.431743 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:13.431982 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.432835 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.433923 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.434381 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.434527 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.435253 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:13.435257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:13.435258 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:13.435468 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:13.435512 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:13.435526 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:13.435597 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.435658 (+ 61us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.435737 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260810 23:07:13.437172 13634 raft_consensus.cc:348] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d0f633d50437b9b8c4b2dfa4b4444" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46205 } }
I20260810 23:07:13.437325 13634 raft_consensus.cc:374] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:13.437348 13634 raft_consensus.cc:729] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c81d0f633d50437b9b8c4b2dfa4b4444, State: Initialized, Role: FOLLOWER
I20260810 23:07:13.437496 13634 consensus_queue.cc:260] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d0f633d50437b9b8c4b2dfa4b4444" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46205 } }
I20260810 23:07:13.437558 13634 raft_consensus.cc:388] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:13.437600 13634 raft_consensus.cc:482] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:13.437649 13634 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:13.438221 13634 raft_consensus.cc:504] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d0f633d50437b9b8c4b2dfa4b4444" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46205 } }
I20260810 23:07:13.438334 13634 leader_election.cc:302] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c81d0f633d50437b9b8c4b2dfa4b4444; no voters:
I20260810 23:07:13.438524 13634 leader_election.cc:288] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:13.438591 13642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:13.438731 13642 raft_consensus.cc:686] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [term 1 LEADER]: Becoming Leader. State: Replica: c81d0f633d50437b9b8c4b2dfa4b4444, State: Running, Role: LEADER
I20260810 23:07:13.438776 13634 sys_catalog.cc:564] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:13.438820 13642 consensus_queue.cc:237] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d0f633d50437b9b8c4b2dfa4b4444" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46205 } }
I20260810 23:07:13.439211 13643 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c81d0f633d50437b9b8c4b2dfa4b4444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d0f633d50437b9b8c4b2dfa4b4444" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46205 } } }
I20260810 23:07:13.439220 13644 sys_catalog.cc:455] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c81d0f633d50437b9b8c4b2dfa4b4444. Latest consensus state: current_term: 1 leader_uuid: "c81d0f633d50437b9b8c4b2dfa4b4444" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c81d0f633d50437b9b8c4b2dfa4b4444" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 46205 } } }
I20260810 23:07:13.439425 13644 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:13.439582 13643 sys_catalog.cc:458] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:13.439687 13651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:13.440174 13651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:13.441640 13651 catalog_manager.cc:1347] Generated new cluster ID: 1e132c08f9224aaa8f556c6bf1b0283b
I20260810 23:07:13.441696 13651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:13.448845 13651 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:13.449487 13651 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:13.461414 13651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444: Generated new TSK 0
I20260810 23:07:13.461597 13651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:13.466277 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.65:0
--local_ip_for_outbound_sockets=127.23.56.65
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:46205
--builtin_ntp_servers=127.23.56.84:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:07:13.548794 13661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:13.549024 13661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:13.549062 13661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:13.550199 13661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:13.550258 13661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:13.550302 13661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.65
I20260810 23:07:13.551739 13661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.56.65
--webserver_port=0
--tserver_master_addrs=127.23.56.126:46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.65
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:13.551998 13661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:13.552292 13661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:13.558144 13668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.558251 13670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.558152 13667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.558591 13661 server_base.cc:1034] running on GCE node
I20260810 23:07:13.558848 13661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:13.559116 13661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:13.560303 13661 hybrid_clock.cc:648] HybridClock initialized: now 1786403233560279 us; error 43 us; skew 500 ppm
I20260810 23:07:13.561614 13661 webserver.cc:460] Webserver started at http://127.23.56.65:35683/ using document root <none> and password file <none>
I20260810 23:07:13.561857 13661 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:13.561924 13661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:13.562067 13661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:13.563112 13661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/data/instance:
uuid: "ab69c09089f242cb80432a2535577f73"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "96162076353c91e8e3be280f8e0e4268"
server_key_iv: "2f56d3752457319a5be183430f4a3590"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.563551 13661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance:
uuid: "ab69c09089f242cb80432a2535577f73"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "96162076353c91e8e3be280f8e0e4268"
server_key_iv: "2f56d3752457319a5be183430f4a3590"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.564867 13661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:07:13.565621 13677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.565778 13661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:13.565865 13661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/wal
uuid: "ab69c09089f242cb80432a2535577f73"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "96162076353c91e8e3be280f8e0e4268"
server_key_iv: "2f56d3752457319a5be183430f4a3590"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.565918 13661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:13.573715 13661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:13.573951 13661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:13.574049 13661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:13.574213 13661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:13.574473 13661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:13.574513 13661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.574534 13661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:13.574546 13661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.580300 13661 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.65:44437
I20260810 23:07:13.580353 13804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.65:44437 every 8 connection(s)
I20260810 23:07:13.580678 13661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/data/info.pb
I20260810 23:07:13.581727 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 13661
I20260810 23:07:13.581832 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-0/wal/instance
I20260810 23:07:13.582048 23777 external_mini_cluster.cc:1428] Setting key bc3c0a5c1f16bbc2c9940225a4246842
I20260810 23:07:13.583168 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.66:0
--local_ip_for_outbound_sockets=127.23.56.66
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:46205
--builtin_ntp_servers=127.23.56.84:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 23:07:13.586812 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.581371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:38489 (local address 127.23.56.126:46205)
0810 23:07:13.581436 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.581439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.581686 (+ 247us) server_negotiation.cc:407] Connection header received
0810 23:07:13.582437 (+ 751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.582438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.582455 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.582518 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:13.583390 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.583890 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.585238 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.585559 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.585636 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.586481 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:13.586483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:13.586484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:13.586489 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:13.586507 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:13.586512 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:13.586576 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.586660 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.586691 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260810 23:07:13.586856 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.581324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46205 (local address 127.23.56.65:38489)
0810 23:07:13.581516 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.581573 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.582269 (+ 696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.582569 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.582572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.582580 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:13.583306 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.583313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.583988 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.583989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.585147 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.585150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.585646 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.585646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.585749 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.586329 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:13.586329 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:13.586358 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:13.586380 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:13.586577 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:13.586580 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:13.586582 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.586624 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.586702 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260810 23:07:13.587515 13805 heartbeater.cc:347] Connected to a master server at 127.23.56.126:46205
I20260810 23:07:13.587632 13805 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:13.587824 13805 heartbeater.cc:511] Master 127.23.56.126:46205 requested a full tablet report, sending...
I20260810 23:07:13.588585 13592 ts_manager.cc:194] Registered new tserver with Master: ab69c09089f242cb80432a2535577f73 (127.23.56.65:44437)
I20260810 23:07:13.589700 13592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.65:38489
W20260810 23:07:13.666479 13808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:13.666749 13808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:13.666774 13808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:13.667948 13808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:13.667985 13808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:13.668011 13808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.66
I20260810 23:07:13.669366 13808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.56.66
--webserver_port=0
--tserver_master_addrs=127.23.56.126:46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.66
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:13.669549 13808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:13.669786 13808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:13.674597 13814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.674567 13818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.674583 13816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.674654 13808 server_base.cc:1034] running on GCE node
I20260810 23:07:13.674988 13808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:13.675278 13808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:13.676446 13808 hybrid_clock.cc:648] HybridClock initialized: now 1786403233676433 us; error 30 us; skew 500 ppm
I20260810 23:07:13.677621 13808 webserver.cc:460] Webserver started at http://127.23.56.66:36929/ using document root <none> and password file <none>
I20260810 23:07:13.677868 13808 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:13.677935 13808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:13.678082 13808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:13.679116 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/data/instance:
uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "df56d115a42156899697ffaf05f6d735"
server_key_iv: "86b627f4f2bba5f360500b3a7d552330"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.679562 13808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance:
uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "df56d115a42156899697ffaf05f6d735"
server_key_iv: "86b627f4f2bba5f360500b3a7d552330"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.680845 13808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:13.681547 13826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.681722 13808 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:13.681818 13808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/wal
uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "df56d115a42156899697ffaf05f6d735"
server_key_iv: "86b627f4f2bba5f360500b3a7d552330"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.681897 13808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:13.694223 13808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:13.694490 13808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:13.694633 13808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:13.694854 13808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:13.695209 13808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:13.695266 13808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.695305 13808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:13.695331 13808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.701225 13808 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.66:36655
I20260810 23:07:13.701260 13955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.66:36655 every 8 connection(s)
I20260810 23:07:13.701591 13808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/data/info.pb
I20260810 23:07:13.706358 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.702308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.66:55471 (local address 127.23.56.126:46205)
0810 23:07:13.702393 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.702396 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.702586 (+ 190us) server_negotiation.cc:407] Connection header received
0810 23:07:13.703018 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.703019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.703029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.703079 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:13.703735 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.704219 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.705160 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.705337 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.705386 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.706155 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:13.706156 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:13.706156 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:13.706160 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:13.706170 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:13.706174 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:13.706213 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.706282 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.706300 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 23:07:13.706542 13958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.702274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46205 (local address 127.23.56.66:55471)
0810 23:07:13.702471 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.702503 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.702942 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.703174 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.703178 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.703189 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:13.703662 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.703666 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.704295 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.704296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.705087 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.705089 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.705408 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.705410 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.705512 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.706036 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:13.706037 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:13.706063 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:13.706084 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:13.706223 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:13.706225 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:13.706227 (+ 2us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.706272 (+ 45us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.706342 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":97,"threads_started":1}
I20260810 23:07:13.707011 13956 heartbeater.cc:347] Connected to a master server at 127.23.56.126:46205
I20260810 23:07:13.707085 13956 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:13.707331 13956 heartbeater.cc:511] Master 127.23.56.126:46205 requested a full tablet report, sending...
I20260810 23:07:13.707705 13592 ts_manager.cc:194] Registered new tserver with Master: 0d06cab2f5864eddb3a1bfb5c9115dfc (127.23.56.66:36655)
I20260810 23:07:13.708076 13592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.66:55471
I20260810 23:07:13.709052 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 13808
I20260810 23:07:13.709158 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/wal/instance
I20260810 23:07:13.709373 23777 external_mini_cluster.cc:1428] Setting key f57cfb3f8e0b7ca3bcbdd5852fdcfd1f
I20260810 23:07:13.710203 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.56.67:0
--local_ip_for_outbound_sockets=127.23.56.67
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:46205
--builtin_ntp_servers=127.23.56.84:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 23:07:13.792032 13959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:13.792244 13959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:13.792268 13959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:13.793465 13959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:13.793510 13959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:13.793573 13959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.56.67
I20260810 23:07:13.795075 13959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.56.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.56.67
--webserver_port=0
--tserver_master_addrs=127.23.56.126:46205
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.56.67
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:13.795372 13959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:13.795677 13959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:13.800638 13968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:13.800678 13964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.801015 13959 server_base.cc:1034] running on GCE node
W20260810 23:07:13.800673 13966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:13.801904 13959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:13.802103 13959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:13.803265 13959 hybrid_clock.cc:648] HybridClock initialized: now 1786403233803256 us; error 40 us; skew 500 ppm
I20260810 23:07:13.804472 13959 webserver.cc:460] Webserver started at http://127.23.56.67:37043/ using document root <none> and password file <none>
I20260810 23:07:13.804737 13959 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:13.804808 13959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:13.804960 13959 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:13.805991 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/data/instance:
uuid: "deaee8126e434f15a6c38d43da8a67b8"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "0a04f4fd0ed002743844eeda9b551399"
server_key_iv: "aa15d13a25654587b2e9b87882a979f0"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.806352 13959 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance:
uuid: "deaee8126e434f15a6c38d43da8a67b8"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "0a04f4fd0ed002743844eeda9b551399"
server_key_iv: "aa15d13a25654587b2e9b87882a979f0"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.807760 13959 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:13.808637 13974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.808897 13959 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:13.809007 13959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/wal
uuid: "deaee8126e434f15a6c38d43da8a67b8"
format_stamp: "Formatted at 2026-08-10 23:07:13 on dist-test-slave-1xrh"
server_key: "0a04f4fd0ed002743844eeda9b551399"
server_key_iv: "aa15d13a25654587b2e9b87882a979f0"
server_key_version: "encryptionkey@0"
I20260810 23:07:13.809108 13959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:13.825016 13959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:13.825347 13959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:13.825516 13959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:13.825766 13959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 23:07:13.826109 13959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 23:07:13.826169 13959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.826213 13959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 23:07:13.826248 13959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:13.832352 13959 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.67:41581
I20260810 23:07:13.832391 14105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.67:41581 every 8 connection(s)
I20260810 23:07:13.832751 13959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/data/info.pb
I20260810 23:07:13.835608 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 13959
I20260810 23:07:13.835731 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/wal/instance
I20260810 23:07:13.835937 23777 external_mini_cluster.cc:1428] Setting key 202eded724fa285e126ec4f0b17f39b3
I20260810 23:07:13.838346 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.833601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.67:45071 (local address 127.23.56.126:46205)
0810 23:07:13.833686 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.833690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.833897 (+ 207us) server_negotiation.cc:407] Connection header received
0810 23:07:13.834361 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.834362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.834375 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.834444 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:13.835089 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.835610 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.836995 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.837181 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.837236 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.838096 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:13.838097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:13.838098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:13.838103 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:13.838116 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:13.838120 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:13.838170 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.838268 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.838288 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:07:13.838527 14108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.833571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:46205 (local address 127.23.56.67:45071)
0810 23:07:13.833774 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.833808 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.834267 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.834495 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.834498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.834506 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:13.834984 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.834988 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.835711 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.835713 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.836919 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.836921 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.837252 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.837253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.837355 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.837922 (+ 567us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 23:07:13.837922 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:13.837958 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 23:07:13.837979 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:13.838184 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:13.838189 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:13.838192 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.838258 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.838333 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":108,"threads_started":1}
I20260810 23:07:13.839026 14106 heartbeater.cc:347] Connected to a master server at 127.23.56.126:46205
I20260810 23:07:13.839102 14106 heartbeater.cc:464] Registering TS with master...
I20260810 23:07:13.839341 14106 heartbeater.cc:511] Master 127.23.56.126:46205 requested a full tablet report, sending...
I20260810 23:07:13.839689 13592 ts_manager.cc:194] Registered new tserver with Master: deaee8126e434f15a6c38d43da8a67b8 (127.23.56.67:41581)
I20260810 23:07:13.840095 13592 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.56.67:45071
I20260810 23:07:13.847543 23777 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 23:07:13.851500 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.848912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55674 (local address 127.23.56.126:46205)
0810 23:07:13.848974 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.848976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.849079 (+ 103us) server_negotiation.cc:407] Connection header received
0810 23:07:13.849116 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.849117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.849125 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.849172 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:13.849293 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.849741 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.850419 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.850596 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.850642 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.851267 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:13.851268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:13.851268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:13.851272 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:13.851286 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:13.851290 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:13.851333 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.851396 (+ 63us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.851415 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:07:13.852772 13592 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 23:07:13.854278 13592 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 23:07:13.860605 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.856740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41581 (local address 127.0.0.1:49758)
0810 23:07:13.856869 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.856902 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.856964 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.857274 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.857275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.857277 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.857333 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.857335 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.858322 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.858323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.859605 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.859608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.860312 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.860313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.860390 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.860404 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.860487 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.860535 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260810 23:07:13.860616 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.856817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49758 (local address 127.23.56.67:41581)
0810 23:07:13.857090 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.857094 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.857102 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:13.857158 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.857160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.857175 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.857422 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.857471 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.858232 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.859695 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.860221 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.860310 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.860397 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.860500 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.860535 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":89,"threads_started":1}
I20260810 23:07:13.860947 14115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.856546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:44437 (local address 127.0.0.1:52424)
0810 23:07:13.856852 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.856894 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.856964 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.857542 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.857543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.857545 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.857576 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.857578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.858794 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.858795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.859741 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.859743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.860673 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.860674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.860758 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.860762 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.860814 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.860850 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":55,"threads_started":1}
I20260810 23:07:13.861063 14119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.856546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:36655 (local address 127.0.0.1:43290)
0810 23:07:13.856852 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.856894 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.856965 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.857211 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.857214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.857216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.857277 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.857278 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.858577 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.858578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.859654 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.859656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.860796 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.860796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.860851 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.860854 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.860898 (+ 44us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.860929 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":3,"thread_start_us":87,"threads_started":1}
I20260810 23:07:13.861261 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.856663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43290 (local address 127.23.56.66:36655)
0810 23:07:13.857032 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.857037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.857050 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:07:13.857092 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.857094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.857112 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.857196 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.857364 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.858486 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.859739 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.860477 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.860551 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.860634 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.860988 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.861184 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":88,"threads_started":1}
I20260810 23:07:13.861254 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.856662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.23.56.65:44437)
0810 23:07:13.857331 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.857336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.857346 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:13.857383 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.857385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.857399 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.857471 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.857678 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.858720 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.859827 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.860491 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.860551 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.860634 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.861122 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.861174 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":91,"threads_started":1}
I20260810 23:07:13.861346 14030 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa34ff928e04fbda56cccc9d74358df (DEFAULT_TABLE table=test-table [id=e94946856b3640a191c2e3f810670f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:13.861735 14030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa34ff928e04fbda56cccc9d74358df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:13.861858 13729 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa34ff928e04fbda56cccc9d74358df (DEFAULT_TABLE table=test-table [id=e94946856b3640a191c2e3f810670f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:13.862105 13729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa34ff928e04fbda56cccc9d74358df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:13.863862 13877 tablet_service.cc:1467] Processing CreateTablet for tablet 3aa34ff928e04fbda56cccc9d74358df (DEFAULT_TABLE table=test-table [id=e94946856b3640a191c2e3f810670f03]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 23:07:13.864195 13877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3aa34ff928e04fbda56cccc9d74358df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:13.864653 14121 tablet_bootstrap.cc:492] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: Bootstrap starting.
I20260810 23:07:13.864949 14122 tablet_bootstrap.cc:492] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73: Bootstrap starting.
I20260810 23:07:13.865393 14121 tablet_bootstrap.cc:654] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:13.865628 14122 tablet_bootstrap.cc:654] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:13.865764 14121 log.cc:824] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:13.865994 14122 log.cc:824] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:13.866581 14121 tablet_bootstrap.cc:492] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: No bootstrap required, opened a new log
I20260810 23:07:13.866658 14121 ts_tablet_manager.cc:1397] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:07:13.866746 14122 tablet_bootstrap.cc:492] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73: No bootstrap required, opened a new log
I20260810 23:07:13.866813 14122 ts_tablet_manager.cc:1397] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:07:13.867758 14125 tablet_bootstrap.cc:492] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: Bootstrap starting.
I20260810 23:07:13.868456 14125 tablet_bootstrap.cc:654] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:13.868770 14125 log.cc:824] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:13.869171 14121 raft_consensus.cc:348] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } }
I20260810 23:07:13.869349 14121 raft_consensus.cc:374] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:13.869395 14121 raft_consensus.cc:729] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deaee8126e434f15a6c38d43da8a67b8, State: Initialized, Role: FOLLOWER
I20260810 23:07:13.869530 14125 tablet_bootstrap.cc:492] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: No bootstrap required, opened a new log
I20260810 23:07:13.869441 14122 raft_consensus.cc:348] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } }
I20260810 23:07:13.869513 14121 consensus_queue.cc:260] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } }
I20260810 23:07:13.869585 14122 raft_consensus.cc:374] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:13.869598 14125 ts_tablet_manager.cc:1397] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 23:07:13.869632 14122 raft_consensus.cc:729] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab69c09089f242cb80432a2535577f73, State: Initialized, Role: FOLLOWER
I20260810 23:07:13.869758 14121 ts_tablet_manager.cc:1428] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 23:07:13.869750 14122 consensus_queue.cc:260] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } }
I20260810 23:07:13.869889 14106 heartbeater.cc:503] Master 127.23.56.126:46205 was elected leader, sending a full tablet report...
I20260810 23:07:13.869992 14122 ts_tablet_manager.cc:1428] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 23:07:13.870029 13805 heartbeater.cc:503] Master 127.23.56.126:46205 was elected leader, sending a full tablet report...
I20260810 23:07:13.871958 14125 raft_consensus.cc:348] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } }
I20260810 23:07:13.872074 14125 raft_consensus.cc:374] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:13.872095 14125 raft_consensus.cc:729] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d06cab2f5864eddb3a1bfb5c9115dfc, State: Initialized, Role: FOLLOWER
I20260810 23:07:13.872164 14125 consensus_queue.cc:260] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } }
I20260810 23:07:13.872334 14125 ts_tablet_manager.cc:1428] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260810 23:07:13.872399 13956 heartbeater.cc:503] Master 127.23.56.126:46205 was elected leader, sending a full tablet report...
I20260810 23:07:13.950083 14128 raft_consensus.cc:482] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 23:07:13.950210 14128 raft_consensus.cc:504] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } }
I20260810 23:07:13.950614 14128 leader_election.cc:288] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d06cab2f5864eddb3a1bfb5c9115dfc (127.23.56.66:36655), deaee8126e434f15a6c38d43da8a67b8 (127.23.56.67:41581)
W20260810 23:07:13.952176 13957 tablet.cc:2369] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 23:07:13.953869 14120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.951128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:38735 (local address 127.23.56.67:41581)
0810 23:07:13.951207 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.951209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.951218 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:07:13.951270 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.951271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.951281 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.951329 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.951458 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.952126 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.953151 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.953518 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.953605 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.953614 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.953755 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.953791 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 23:07:13.953902 14117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.950932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.56.65:46999 (local address 127.23.56.66:36655)
0810 23:07:13.950981 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.950983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.950992 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:07:13.951025 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.951026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.951035 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.951086 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 23:07:13.951291 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.952089 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.953163 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.953538 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.953632 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.953641 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.953785 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.953834 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260810 23:07:13.953907 13807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.950796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.66:36655 (local address 127.23.56.65:46999)
0810 23:07:13.950870 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.950891 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.950962 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.951107 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.951108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.951110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.951207 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.951210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.952197 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.952198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.953060 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.953063 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.953648 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.953649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.953710 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.953713 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.953779 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.953822 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 23:07:13.953920 14130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.950921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.67:41581 (local address 127.23.56.65:38735)
0810 23:07:13.951112 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.951122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:13.951197 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:13.951340 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:13.951341 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:13.951342 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 23:07:13.951390 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.951391 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.952206 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.952208 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.953055 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:13.953057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.953606 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.953607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:13.953680 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.953683 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 23:07:13.953750 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 23:07:13.953788 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":110,"threads_started":1}
I20260810 23:07:13.954335 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa34ff928e04fbda56cccc9d74358df" candidate_uuid: "ab69c09089f242cb80432a2535577f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" is_pre_election: true
I20260810 23:07:13.954329 14055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa34ff928e04fbda56cccc9d74358df" candidate_uuid: "ab69c09089f242cb80432a2535577f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deaee8126e434f15a6c38d43da8a67b8" is_pre_election: true
I20260810 23:07:13.954492 14055 raft_consensus.cc:2454] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab69c09089f242cb80432a2535577f73 in term 0.
I20260810 23:07:13.954492 13902 raft_consensus.cc:2454] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab69c09089f242cb80432a2535577f73 in term 0.
I20260810 23:07:13.954725 13681 leader_election.cc:302] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d06cab2f5864eddb3a1bfb5c9115dfc, ab69c09089f242cb80432a2535577f73; no voters:
I20260810 23:07:13.954924 14128 raft_consensus.cc:2781] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 23:07:13.954986 14128 raft_consensus.cc:482] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 23:07:13.955011 14128 raft_consensus.cc:3037] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:13.955767 14128 raft_consensus.cc:504] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } }
I20260810 23:07:13.955919 14128 leader_election.cc:288] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [CANDIDATE]: Term 1 election: Requested vote from peers 0d06cab2f5864eddb3a1bfb5c9115dfc (127.23.56.66:36655), deaee8126e434f15a6c38d43da8a67b8 (127.23.56.67:41581)
I20260810 23:07:13.956125 13902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa34ff928e04fbda56cccc9d74358df" candidate_uuid: "ab69c09089f242cb80432a2535577f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc"
I20260810 23:07:13.956166 14055 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3aa34ff928e04fbda56cccc9d74358df" candidate_uuid: "ab69c09089f242cb80432a2535577f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "deaee8126e434f15a6c38d43da8a67b8"
I20260810 23:07:13.956215 13902 raft_consensus.cc:3037] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:13.956228 14055 raft_consensus.cc:3037] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:13.957134 14055 raft_consensus.cc:2454] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab69c09089f242cb80432a2535577f73 in term 1.
I20260810 23:07:13.957134 13902 raft_consensus.cc:2454] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab69c09089f242cb80432a2535577f73 in term 1.
I20260810 23:07:13.957321 13681 leader_election.cc:302] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab69c09089f242cb80432a2535577f73, deaee8126e434f15a6c38d43da8a67b8; no voters:
I20260810 23:07:13.957427 14128 raft_consensus.cc:2781] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:13.957584 14128 raft_consensus.cc:686] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [term 1 LEADER]: Becoming Leader. State: Replica: ab69c09089f242cb80432a2535577f73, State: Running, Role: LEADER
I20260810 23:07:13.957690 14128 consensus_queue.cc:237] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } }
I20260810 23:07:13.958552 13591 catalog_manager.cc:5495] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab69c09089f242cb80432a2535577f73 (127.23.56.65). New cstate: current_term: 1 leader_uuid: "ab69c09089f242cb80432a2535577f73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab69c09089f242cb80432a2535577f73" member_type: VOTER last_known_addr { host: "127.23.56.65" port: 44437 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 } health_report { overall_health: UNKNOWN } } }
I20260810 23:07:13.978516 14118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.976077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52432 (local address 127.23.56.65:44437)
0810 23:07:13.976137 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:13.976139 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:13.976149 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:13.976222 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:13.976223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:13.976250 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:13.976325 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:13.976453 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.976932 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.977693 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:13.977881 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:13.977974 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:13.978164 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:13.978266 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:13.978356 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:13.978395 (+ 39us) server_negotiation.cc:299] Negotiation successful
0810 23:07:13.978417 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:07:13.980521 14055 raft_consensus.cc:1261] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [term 1 FOLLOWER]: Refusing update from remote peer ab69c09089f242cb80432a2535577f73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:13.980521 13902 raft_consensus.cc:1261] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [term 1 FOLLOWER]: Refusing update from remote peer ab69c09089f242cb80432a2535577f73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:13.980944 14131 consensus_queue.cc:1035] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "deaee8126e434f15a6c38d43da8a67b8" member_type: VOTER last_known_addr { host: "127.23.56.67" port: 41581 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:13.981119 14128 consensus_queue.cc:1035] T 3aa34ff928e04fbda56cccc9d74358df P ab69c09089f242cb80432a2535577f73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d06cab2f5864eddb3a1bfb5c9115dfc" member_type: VOTER last_known_addr { host: "127.23.56.66" port: 36655 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:13.983036 14135 mvcc.cc:205] Tried to move back new op lower bound from 7317107646380822528 to 7317107646291378176. Current Snapshot: MvccSnapshot[applied={T|T < 7317107646380822528}]
I20260810 23:07:13.983454 14136 mvcc.cc:205] Tried to move back new op lower bound from 7317107646380822528 to 7317107646291378176. Current Snapshot: MvccSnapshot[applied={T|T < 7317107646380822528}]
I20260810 23:07:13.985297 14138 mvcc.cc:205] Tried to move back new op lower bound from 7317107646380822528 to 7317107646291378176. Current Snapshot: MvccSnapshot[applied={T|T < 7317107646380822528 or (T in {7317107646380822528})}]
I20260810 23:07:13.985858 13592 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:13.985947 13592 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 23:07:13.986654 13592 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c81d0f633d50437b9b8c4b2dfa4b4444: Sending DeleteTablet for 3 replicas of tablet 3aa34ff928e04fbda56cccc9d74358df
I20260810 23:07:13.987001 13877 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa34ff928e04fbda56cccc9d74358df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:13 UTC) from {username='slave'} at 127.0.0.1:43290
I20260810 23:07:13.987025 14030 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa34ff928e04fbda56cccc9d74358df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:13 UTC) from {username='slave'} at 127.0.0.1:49758
I20260810 23:07:13.987033 13729 tablet_service.cc:1514] Processing DeleteTablet for tablet 3aa34ff928e04fbda56cccc9d74358df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 23:07:13 UTC) from {username='slave'} at 127.0.0.1:52424
I20260810 23:07:13.987242 14144 tablet_replica.cc:330] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: stopping tablet replica
I20260810 23:07:13.987476 14145 tablet_replica.cc:330] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: stopping tablet replica
I20260810 23:07:13.987545 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 13661
I20260810 23:07:13.987331 14144 raft_consensus.cc:2229] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:13.988831 14144 raft_consensus.cc:2258] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:13.987572 14145 raft_consensus.cc:2229] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 23:07:13.989033 14145 raft_consensus.cc:2258] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 23:07:13.989878 14145 ts_tablet_manager.cc:1905] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:13.990010 14144 ts_tablet_manager.cc:1905] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 23:07:13.991750 14145 ts_tablet_manager.cc:1918] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:13.991837 14145 log.cc:1196] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-1/wal/wals/3aa34ff928e04fbda56cccc9d74358df
I20260810 23:07:13.991850 14144 ts_tablet_manager.cc:1918] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 23:07:13.991958 14144 log.cc:1196] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: Deleting WAL directory at /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786402992661514-23777-0/minicluster-data/ts-2/wal/wals/3aa34ff928e04fbda56cccc9d74358df
I20260810 23:07:13.992110 14145 ts_tablet_manager.cc:1939] T 3aa34ff928e04fbda56cccc9d74358df P 0d06cab2f5864eddb3a1bfb5c9115dfc: Deleting consensus metadata
I20260810 23:07:13.992214 14144 ts_tablet_manager.cc:1939] T 3aa34ff928e04fbda56cccc9d74358df P deaee8126e434f15a6c38d43da8a67b8: Deleting consensus metadata
I20260810 23:07:13.992530 13579 catalog_manager.cc:4841] TS 0d06cab2f5864eddb3a1bfb5c9115dfc (127.23.56.66:36655): tablet 3aa34ff928e04fbda56cccc9d74358df (table test-table [id=e94946856b3640a191c2e3f810670f03]) successfully deleted
I20260810 23:07:13.992743 13579 catalog_manager.cc:4841] TS deaee8126e434f15a6c38d43da8a67b8 (127.23.56.67:41581): tablet 3aa34ff928e04fbda56cccc9d74358df (table test-table [id=e94946856b3640a191c2e3f810670f03]) successfully deleted
W20260810 23:07:13.995918 13576 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.56.65:44437 (error 108)
I20260810 23:07:13.996142 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 13808
I20260810 23:07:13.996450 14116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:13.996320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.65:44437 (local address 127.0.0.1:52444)
0810 23:07:13.996368 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:13.996405 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.65:44437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 23:07:13.996549 13576 catalog_manager.cc:4584] TS ab69c09089f242cb80432a2535577f73 (127.23.56.65:44437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3aa34ff928e04fbda56cccc9d74358df: Network error: Client connection negotiation failed: client connection to 127.23.56.65:44437: connect: Connection refused (error 111)
I20260810 23:07:14.004539 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 13959
I20260810 23:07:14.011382 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 13560
2026-08-10T23:07:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (734 ms)
[----------] 6 tests from EncryptionPolicyTest (8438 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:14 dist-test-slave-1xrh krb5kdc[14151](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:14 dist-test-slave-1xrh krb5kdc[14151](info): set up 2 sockets
Aug 10 23:07:14 dist-test-slave-1xrh krb5kdc[14151](info): commencing operation
krb5kdc: starting...
W20260810 23:07:15.095997 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:15 dist-test-slave-1xrh krb5kdc[14151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:15.216215 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:15.298507 14165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:15.298717 14165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:15.298740 14165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:15.299860 14165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:15.299893 14165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:15.299907 14165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:15.299921 14165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:15.299932 14165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:15.301323 14165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41057
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:15.301510 14165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:15.301688 14165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:15.306424 14170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:15.306432 14171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:15.306435 14173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:15.306770 14165 server_base.cc:1034] running on GCE node
I20260810 23:07:15.307005 14165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:15.307333 14165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:15.308475 14165 hybrid_clock.cc:648] HybridClock initialized: now 1786403235308456 us; error 40 us; skew 500 ppm
Aug 10 23:07:15 dist-test-slave-1xrh krb5kdc[14151](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:15.311738 14165 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:15.312330 14165 webserver.cc:460] Webserver started at http://127.0.0.1:37001/ using document root <none> and password file <none>
I20260810 23:07:15.312592 14165 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:15.312660 14165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:15.312800 14165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:15.314231 14165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "a550bf4625af4356976bb22650545abb"
format_stamp: "Formatted at 2026-08-10 23:07:15 on dist-test-slave-1xrh"
server_key: "0c673d073346fe550cf0756f7d12ae15"
server_key_iv: "a8a35e7f5650c5dd7f433d382d80e510"
server_key_version: "encryptionkey@0"
I20260810 23:07:15.314592 14165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "a550bf4625af4356976bb22650545abb"
format_stamp: "Formatted at 2026-08-10 23:07:15 on dist-test-slave-1xrh"
server_key: "0c673d073346fe550cf0756f7d12ae15"
server_key_iv: "a8a35e7f5650c5dd7f433d382d80e510"
server_key_version: "encryptionkey@0"
I20260810 23:07:15.316304 14165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260810 23:07:15.317327 14181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:15.317523 14165 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 23:07:15.317615 14165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "a550bf4625af4356976bb22650545abb"
format_stamp: "Formatted at 2026-08-10 23:07:15 on dist-test-slave-1xrh"
server_key: "0c673d073346fe550cf0756f7d12ae15"
server_key_iv: "a8a35e7f5650c5dd7f433d382d80e510"
server_key_version: "encryptionkey@0"
I20260810 23:07:15.317703 14165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:15.324486 14165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:15.326725 14165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:15.326891 14165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:15.330964 14165 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41057
I20260810 23:07:15.331025 14239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41057 every 8 connection(s)
I20260810 23:07:15.331388 14165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:15.332235 14240 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:15.334408 14240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb: Bootstrap starting.
I20260810 23:07:15.334975 14240 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:15.335304 14240 log.cc:824] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:15.336015 14240 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb: No bootstrap required, opened a new log
I20260810 23:07:15.337553 14240 raft_consensus.cc:348] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a550bf4625af4356976bb22650545abb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260810 23:07:15.337684 14240 raft_consensus.cc:374] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:15.337707 14240 raft_consensus.cc:729] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a550bf4625af4356976bb22650545abb, State: Initialized, Role: FOLLOWER
I20260810 23:07:15.337883 14240 consensus_queue.cc:260] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a550bf4625af4356976bb22650545abb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260810 23:07:15.337944 14240 raft_consensus.cc:388] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:15.337965 14240 raft_consensus.cc:482] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:15.338025 14240 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:15.338552 14240 raft_consensus.cc:504] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a550bf4625af4356976bb22650545abb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260810 23:07:15.338672 14240 leader_election.cc:302] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a550bf4625af4356976bb22650545abb; no voters:
I20260810 23:07:15.338843 14240 leader_election.cc:288] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:15.338881 14243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:15.339051 14243 raft_consensus.cc:686] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [term 1 LEADER]: Becoming Leader. State: Replica: a550bf4625af4356976bb22650545abb, State: Running, Role: LEADER
I20260810 23:07:15.339095 14240 sys_catalog.cc:564] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:15.339174 14243 consensus_queue.cc:237] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a550bf4625af4356976bb22650545abb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } }
I20260810 23:07:15.339732 14245 sys_catalog.cc:455] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a550bf4625af4356976bb22650545abb. Latest consensus state: current_term: 1 leader_uuid: "a550bf4625af4356976bb22650545abb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a550bf4625af4356976bb22650545abb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } } }
I20260810 23:07:15.339828 14245 sys_catalog.cc:458] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:15.339994 14244 sys_catalog.cc:455] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a550bf4625af4356976bb22650545abb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a550bf4625af4356976bb22650545abb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41057 } } }
I20260810 23:07:15.340056 14244 sys_catalog.cc:458] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:15.340101 14252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:15.340518 14252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:15.341054 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 14165
I20260810 23:07:15.341180 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:15.341383 23777 external_mini_cluster.cc:1428] Setting key 264d172d196cd47f26da5f455738843f
I20260810 23:07:15.343547 14252 catalog_manager.cc:1347] Generated new cluster ID: e8c40019f235444985b4e70a1e7407f3
I20260810 23:07:15.343606 14252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:07:15 dist-test-slave-1xrh krb5kdc[14151](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:07:15.352811 14252 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:15.353305 14252 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:15.354825 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:15.341832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51407 (local address 127.0.0.1:41057)
0810 23:07:15.343832 (+ 2000us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:15.343841 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:15.343870 (+ 29us) server_negotiation.cc:407] Connection header received
0810 23:07:15.343997 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:15.344004 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:15.344053 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:15.344168 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:15.344965 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:15.346700 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:15.347799 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:15.348410 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:15.348507 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:15.351288 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:15.351292 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:15.351293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:15.351305 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:15.353198 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:15.353632 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:15.353635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:15.353635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:15.353751 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:15.353982 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:15.353983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:15.353984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:15.354240 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:15.354384 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:15.354491 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 23:07:15.354589 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1928,"thread_start_us":68,"threads_started":1}
I20260810 23:07:15.356065 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:15.362746 14252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a550bf4625af4356976bb22650545abb: Generated new TSK 0
I20260810 23:07:15.363005 14252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:15.368929 14264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:15.357553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57807 (local address 127.0.0.1:41057)
0810 23:07:15.357585 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:15.357587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:15.362844 (+ 5257us) server_negotiation.cc:407] Connection header received
0810 23:07:15.362904 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:15.362905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:15.362915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:15.362959 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:15.363632 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:15.364149 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:15.364865 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:15.365093 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:15.365178 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:15.366358 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:15.366360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:15.366360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:15.366367 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:15.367771 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:15.368165 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:15.368166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:15.368167 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:15.368225 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:15.368449 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:15.368450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:15.368450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:15.368551 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:15.368669 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:15.368779 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 23:07:15.368839 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:07:15.370947 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 14165
W20260810 23:07:15.381299 14149 connection.cc:535] client connection to 127.0.0.1:41057 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41057): Connection reset by peer (error 104)
2026-08-10T23:07:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:15 dist-test-slave-1xrh krb5kdc[14272](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:15 dist-test-slave-1xrh krb5kdc[14272](info): set up 2 sockets
Aug 10 23:07:15 dist-test-slave-1xrh krb5kdc[14272](info): commencing operation
krb5kdc: starting...
W20260810 23:07:16.468751 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:16 dist-test-slave-1xrh krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:16.590034 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33831
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35401
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33831
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:16.671455 14286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:16.671640 14286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:16.671661 14286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:16.672844 14286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:16.672887 14286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:16.672902 14286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:16.672947 14286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:16.672976 14286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:16.674387 14286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35401
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33831
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:16.674638 14286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:16.674853 14286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:16.679529 14292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:16.679548 14293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:16.679701 14286 server_base.cc:1034] running on GCE node
W20260810 23:07:16.679720 14295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:16.680001 14286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:16.680287 14286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:16.681454 14286 hybrid_clock.cc:648] HybridClock initialized: now 1786403236681438 us; error 46 us; skew 500 ppm
Aug 10 23:07:16 dist-test-slave-1xrh krb5kdc[14272](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:16.688706 14286 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:16.689158 14286 webserver.cc:460] Webserver started at http://127.0.0.1:43493/ using document root <none> and password file <none>
I20260810 23:07:16.689391 14286 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:16.689465 14286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:16.689617 14286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:16.690701 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "7919dd20fca84d9b9cb781eafec1bfcd"
format_stamp: "Formatted at 2026-08-10 23:07:16 on dist-test-slave-1xrh"
server_key: "17cdc0e1b535e0210bdbb6b8b12f99a1"
server_key_iv: "8347c43ba7486c24327fb1cbcb02cc4b"
server_key_version: "encryptionkey@0"
I20260810 23:07:16.691087 14286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "7919dd20fca84d9b9cb781eafec1bfcd"
format_stamp: "Formatted at 2026-08-10 23:07:16 on dist-test-slave-1xrh"
server_key: "17cdc0e1b535e0210bdbb6b8b12f99a1"
server_key_iv: "8347c43ba7486c24327fb1cbcb02cc4b"
server_key_version: "encryptionkey@0"
I20260810 23:07:16.692444 14286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:07:16.693192 14302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:16.693367 14286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:16.693465 14286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "7919dd20fca84d9b9cb781eafec1bfcd"
format_stamp: "Formatted at 2026-08-10 23:07:16 on dist-test-slave-1xrh"
server_key: "17cdc0e1b535e0210bdbb6b8b12f99a1"
server_key_iv: "8347c43ba7486c24327fb1cbcb02cc4b"
server_key_version: "encryptionkey@0"
I20260810 23:07:16.693547 14286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:16.701762 14286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:16.703599 14286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:16.703758 14286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:16.707753 14286 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33831
I20260810 23:07:16.707813 14359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33831 every 8 connection(s)
I20260810 23:07:16.708160 14286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:16.709035 14360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:16.711279 14360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd: Bootstrap starting.
I20260810 23:07:16.711838 14360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:16.712134 14360 log.cc:824] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:16.712808 14360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd: No bootstrap required, opened a new log
I20260810 23:07:16.714341 14360 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7919dd20fca84d9b9cb781eafec1bfcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33831 } }
I20260810 23:07:16.714473 14360 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:16.714495 14360 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7919dd20fca84d9b9cb781eafec1bfcd, State: Initialized, Role: FOLLOWER
I20260810 23:07:16.714659 14360 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7919dd20fca84d9b9cb781eafec1bfcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33831 } }
I20260810 23:07:16.714718 14360 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:16.714783 14360 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:16.714840 14360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:16.714919 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 14286
I20260810 23:07:16.715046 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:16.715255 23777 external_mini_cluster.cc:1428] Setting key 3de7eacb9f1fca0b21f19c929b05b38b
I20260810 23:07:16.715543 14360 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7919dd20fca84d9b9cb781eafec1bfcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33831 } }
I20260810 23:07:16.715656 14360 leader_election.cc:302] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7919dd20fca84d9b9cb781eafec1bfcd; no voters:
I20260810 23:07:16.715821 14360 leader_election.cc:288] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:16.715862 14364 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:16.716007 14364 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [term 1 LEADER]: Becoming Leader. State: Replica: 7919dd20fca84d9b9cb781eafec1bfcd, State: Running, Role: LEADER
I20260810 23:07:16.716081 14360 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:16.716107 14364 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7919dd20fca84d9b9cb781eafec1bfcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33831 } }
I20260810 23:07:16.716446 14367 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7919dd20fca84d9b9cb781eafec1bfcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7919dd20fca84d9b9cb781eafec1bfcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33831 } } }
I20260810 23:07:16.716531 14367 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:16.716624 14366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7919dd20fca84d9b9cb781eafec1bfcd. Latest consensus state: current_term: 1 leader_uuid: "7919dd20fca84d9b9cb781eafec1bfcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7919dd20fca84d9b9cb781eafec1bfcd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33831 } } }
I20260810 23:07:16.716720 14366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:16.716760 14373 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:16.717258 14373 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:16.718905 14373 catalog_manager.cc:1347] Generated new cluster ID: 233db6ee6add4194b13b937dc0f67e2f
I20260810 23:07:16.718961 14373 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:16.721248 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:16.715720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47203 (local address 127.0.0.1:33831)
0810 23:07:16.715979 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:16.715988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:16.716020 (+ 32us) server_negotiation.cc:407] Connection header received
0810 23:07:16.716180 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:16.716187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:16.716276 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:16.716799 (+ 523us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:16.716936 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:16.718312 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:16.719369 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:16.719781 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:16.719900 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:16.720608 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:16.720612 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:16.720612 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:16.720785 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:16.720812 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:16.720823 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:16.720917 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:16.721007 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 23:07:16.721035 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":82,"threads_started":1}
I20260810 23:07:16.722204 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:16.726629 14365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:16.723615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43439 (local address 127.0.0.1:33831)
0810 23:07:16.723675 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:16.723678 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:16.723745 (+ 67us) server_negotiation.cc:407] Connection header received
0810 23:07:16.723830 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:16.723831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:16.723840 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:16.723885 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:16.724001 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:16.724720 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:16.725445 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:16.725673 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:16.725745 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:16.726387 (+ 642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:16.726388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:16.726389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:16.726393 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:16.726406 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:16.726410 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:16.726463 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:16.726523 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 23:07:16.726535 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 23:07:16.739059 14373 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:16.739569 14373 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:16.742164 14373 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7919dd20fca84d9b9cb781eafec1bfcd: Generated new TSK 0
I20260810 23:07:16.742317 14373 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:16.748951 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 14286
W20260810 23:07:16.757174 14270 connection.cc:535] client connection to 127.0.0.1:33831 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33831): Connection reset by peer (error 104)
2026-08-10T23:07:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:16 dist-test-slave-1xrh krb5kdc[14396](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:16 dist-test-slave-1xrh krb5kdc[14396](info): set up 2 sockets
Aug 10 23:07:16 dist-test-slave-1xrh krb5kdc[14396](info): commencing operation
krb5kdc: starting...
W20260810 23:07:17.847461 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:17 dist-test-slave-1xrh krb5kdc[14396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:17.965626 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37837
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37837
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:18.047710 14411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:18.047959 14411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:18.048002 14411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:18.049615 14411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:18.049671 14411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:18.049695 14411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:18.049732 14411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:18.049755 14411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:18.051302 14411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37837
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37837
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:18.051563 14411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:18.051808 14411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:18.056519 14419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:18.056546 14417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:18.056627 14411 server_base.cc:1034] running on GCE node
W20260810 23:07:18.056543 14416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:18.056918 14411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:18.057214 14411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:18.058374 14411 hybrid_clock.cc:648] HybridClock initialized: now 1786403238058367 us; error 30 us; skew 500 ppm
Aug 10 23:07:18 dist-test-slave-1xrh krb5kdc[14396](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403238, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:18.060827 14411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:18.061280 14411 webserver.cc:460] Webserver started at http://127.0.0.1:41849/ using document root <none> and password file <none>
I20260810 23:07:18.061502 14411 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:18.061569 14411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:18.061719 14411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:18.062736 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "8d71eb3bf6294c279a6980b01f8590f3"
format_stamp: "Formatted at 2026-08-10 23:07:18 on dist-test-slave-1xrh"
server_key: "6c514c427f05a05d12f7ef1091c30977"
server_key_iv: "18f4c0c7a99211109ac4c4ea8aad2b70"
server_key_version: "encryptionkey@0"
I20260810 23:07:18.063103 14411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "8d71eb3bf6294c279a6980b01f8590f3"
format_stamp: "Formatted at 2026-08-10 23:07:18 on dist-test-slave-1xrh"
server_key: "6c514c427f05a05d12f7ef1091c30977"
server_key_iv: "18f4c0c7a99211109ac4c4ea8aad2b70"
server_key_version: "encryptionkey@0"
I20260810 23:07:18.064446 14411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:07:18.065156 14426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:18.065320 14411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:18.065416 14411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "8d71eb3bf6294c279a6980b01f8590f3"
format_stamp: "Formatted at 2026-08-10 23:07:18 on dist-test-slave-1xrh"
server_key: "6c514c427f05a05d12f7ef1091c30977"
server_key_iv: "18f4c0c7a99211109ac4c4ea8aad2b70"
server_key_version: "encryptionkey@0"
I20260810 23:07:18.065491 14411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:18.068699 14411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:18.070514 14411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:18.070664 14411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:18.074940 14411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37837
I20260810 23:07:18.074973 14484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37837 every 8 connection(s)
I20260810 23:07:18.075361 14411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:18.076309 14485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:18.078585 14485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3: Bootstrap starting.
I20260810 23:07:18.079310 14485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:18.079664 14485 log.cc:824] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:18.080539 14485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3: No bootstrap required, opened a new log
I20260810 23:07:18.080622 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 14411
I20260810 23:07:18.080734 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:18.080937 23777 external_mini_cluster.cc:1428] Setting key 467b6668552f8a7738ddc53abbe9235d
I20260810 23:07:18.082862 14485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d71eb3bf6294c279a6980b01f8590f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } }
I20260810 23:07:18.083027 14485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:18.083065 14485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d71eb3bf6294c279a6980b01f8590f3, State: Initialized, Role: FOLLOWER
I20260810 23:07:18.083245 14485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d71eb3bf6294c279a6980b01f8590f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } }
I20260810 23:07:18.083356 14485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:18.083405 14485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:18.083457 14485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:18.084501 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:18.081449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33633 (local address 127.0.0.1:37837)
0810 23:07:18.081638 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:18.081649 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:18.081684 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:07:18.082831 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:18.082840 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:18.082877 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:18.083024 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:18.083972 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:18.083976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:18.083976 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:18.083981 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:18.084034 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:18.084050 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:18.084123 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:18.084230 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 23:07:18.084280 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":77,"threads_started":1}
I20260810 23:07:18.084542 14485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d71eb3bf6294c279a6980b01f8590f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } }
I20260810 23:07:18.084656 14485 leader_election.cc:302] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d71eb3bf6294c279a6980b01f8590f3; no voters:
I20260810 23:07:18.084820 14485 leader_election.cc:288] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:18.084854 14492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:18.085073 14492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [term 1 LEADER]: Becoming Leader. State: Replica: 8d71eb3bf6294c279a6980b01f8590f3, State: Running, Role: LEADER
I20260810 23:07:18.085093 14485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:18.085258 14492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d71eb3bf6294c279a6980b01f8590f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } }
I20260810 23:07:18.085470 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:18.086885 14494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d71eb3bf6294c279a6980b01f8590f3. Latest consensus state: current_term: 1 leader_uuid: "8d71eb3bf6294c279a6980b01f8590f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d71eb3bf6294c279a6980b01f8590f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } } }
W20260810 23:07:18.087038 14509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 23:07:18.087059 14494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [sys.catalog]: This master's current role is: LEADER
W20260810 23:07:18.087191 14509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:07:18.087137 14493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d71eb3bf6294c279a6980b01f8590f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d71eb3bf6294c279a6980b01f8590f3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37837 } } }
I20260810 23:07:18.087330 14493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:18.087430 14515 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:18.087604 14515 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:18.088940 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:18.087626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56807 (local address 127.0.0.1:37837)
0810 23:07:18.087669 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:18.087671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:18.087848 (+ 177us) server_negotiation.cc:407] Connection header received
0810 23:07:18.087893 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:18.087894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:18.087899 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:18.087937 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:18.088698 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:18.088699 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:18.088700 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:18.088705 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:18.088721 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:18.088726 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:18.088792 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:18.088861 (+ 69us) server_negotiation.cc:299] Negotiation successful
0810 23:07:18.088880 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:07:18.089330 14515 catalog_manager.cc:1347] Generated new cluster ID: f284ff3b87b4458c8d7cc57731389658
I20260810 23:07:18.089382 14515 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:18.113193 14515 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:18.113394 14515 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:18.117049 14515 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d71eb3bf6294c279a6980b01f8590f3: Generated new TSK 0
I20260810 23:07:18.117231 14515 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:18.126349 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 14411
2026-08-10T23:07:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1377 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:18 dist-test-slave-1xrh krb5kdc[14520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:18 dist-test-slave-1xrh krb5kdc[14520](info): set up 2 sockets
Aug 10 23:07:18 dist-test-slave-1xrh krb5kdc[14520](info): commencing operation
krb5kdc: starting...
W20260810 23:07:19.224476 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:19 dist-test-slave-1xrh krb5kdc[14520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:19.349455 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41597
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:19.429857 14534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:19.430083 14534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:19.430123 14534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:19.431268 14534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:19.431334 14534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:19.431360 14534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:19.431393 14534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:19.431421 14534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:19.432891 14534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:19.433166 14534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:19.433389 14534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:19.438194 14542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:19.438220 14534 server_base.cc:1034] running on GCE node
W20260810 23:07:19.438194 14539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:19.438231 14540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:19.438622 14534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:19.438915 14534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:19.440095 14534 hybrid_clock.cc:648] HybridClock initialized: now 1786403239440079 us; error 28 us; skew 500 ppm
Aug 10 23:07:19 dist-test-slave-1xrh krb5kdc[14520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:19.442427 14534 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:19.442835 14534 webserver.cc:460] Webserver started at http://127.0.0.1:33791/ using document root <none> and password file <none>
I20260810 23:07:19.443080 14534 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:19.443173 14534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:19.443318 14534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:19.444365 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "dd2248c5c18c46baaf3796872ecdb458"
format_stamp: "Formatted at 2026-08-10 23:07:19 on dist-test-slave-1xrh"
server_key: "0b6d9604847093d675e675a81f14a0d6"
server_key_iv: "9b6b3dc5502527fdcc05278deaf0bf3c"
server_key_version: "encryptionkey@0"
I20260810 23:07:19.444716 14534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "dd2248c5c18c46baaf3796872ecdb458"
format_stamp: "Formatted at 2026-08-10 23:07:19 on dist-test-slave-1xrh"
server_key: "0b6d9604847093d675e675a81f14a0d6"
server_key_iv: "9b6b3dc5502527fdcc05278deaf0bf3c"
server_key_version: "encryptionkey@0"
I20260810 23:07:19.446012 14534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:19.446702 14549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:19.446857 14534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:19.446969 14534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "dd2248c5c18c46baaf3796872ecdb458"
format_stamp: "Formatted at 2026-08-10 23:07:19 on dist-test-slave-1xrh"
server_key: "0b6d9604847093d675e675a81f14a0d6"
server_key_iv: "9b6b3dc5502527fdcc05278deaf0bf3c"
server_key_version: "encryptionkey@0"
I20260810 23:07:19.447043 14534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:19.456065 14534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:19.457739 14534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:19.457895 14534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:19.461697 14534 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41597
I20260810 23:07:19.461767 14609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41597 every 8 connection(s)
I20260810 23:07:19.462103 14534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:19.462951 14610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:19.463986 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 14534
I20260810 23:07:19.464111 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:19.464293 23777 external_mini_cluster.cc:1428] Setting key 2147bc2eae5ab9fc5fcc5f82353e8afc
I20260810 23:07:19.465579 14610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458: Bootstrap starting.
I20260810 23:07:19.466316 14610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:19.466609 14610 log.cc:824] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:19.467360 14610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458: No bootstrap required, opened a new log
I20260810 23:07:19.469720 14610 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2248c5c18c46baaf3796872ecdb458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } }
I20260810 23:07:19.469889 14610 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:19.469939 14610 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd2248c5c18c46baaf3796872ecdb458, State: Initialized, Role: FOLLOWER
I20260810 23:07:19.470081 14610 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2248c5c18c46baaf3796872ecdb458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } }
I20260810 23:07:19.470168 14610 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:19.470206 14610 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:19.470258 14610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:19.470922 14610 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2248c5c18c46baaf3796872ecdb458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } }
I20260810 23:07:19.471015 14610 leader_election.cc:302] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd2248c5c18c46baaf3796872ecdb458; no voters:
I20260810 23:07:19.471256 14610 leader_election.cc:288] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:19.471274 14616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [term 1 FOLLOWER]: Leader election won for term 1
Aug 10 23:07:19 dist-test-slave-1xrh krb5kdc[14520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:07:19.471392 14616 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [term 1 LEADER]: Becoming Leader. State: Replica: dd2248c5c18c46baaf3796872ecdb458, State: Running, Role: LEADER
I20260810 23:07:19.471485 14616 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2248c5c18c46baaf3796872ecdb458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } }
I20260810 23:07:19.471521 14610 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:19.471828 14616 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd2248c5c18c46baaf3796872ecdb458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2248c5c18c46baaf3796872ecdb458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } } }
I20260810 23:07:19.471900 14616 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:19.472031 14616 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd2248c5c18c46baaf3796872ecdb458. Latest consensus state: current_term: 1 leader_uuid: "dd2248c5c18c46baaf3796872ecdb458" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd2248c5c18c46baaf3796872ecdb458" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41597 } } }
I20260810 23:07:19.472100 14616 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:19.472214 14621 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:19.472733 14621 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:19.474332 14621 catalog_manager.cc:1347] Generated new cluster ID: b48bd7fcb3db4449b7b87462d8e39b52
I20260810 23:07:19.474387 14621 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:19.476560 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:19.464818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53591 (local address 127.0.0.1:41597)
0810 23:07:19.465016 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:19.465026 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:19.465060 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:07:19.465482 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:19.465489 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:19.465533 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:19.465644 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:19.466471 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:19.467515 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:19.468574 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:19.468955 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:19.469036 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:19.471887 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:19.471891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:19.471891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:19.471928 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:19.474990 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:19.475380 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:19.475383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:19.475384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:19.475492 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:19.475715 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:19.475716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:19.475716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:19.475949 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:19.476126 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:19.476205 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 23:07:19.476344 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":112,"threads_started":1}
I20260810 23:07:19.477412 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:19.484064 14621 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:19.484709 14621 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:19.484977 14614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:19.478804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54689 (local address 127.0.0.1:41597)
0810 23:07:19.478841 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:19.478842 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:19.478960 (+ 118us) server_negotiation.cc:407] Connection header received
0810 23:07:19.479033 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:19.479034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:19.479043 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:19.479091 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:19.479635 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:19.480139 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:19.480811 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:19.481036 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:19.481121 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:19.482262 (+ 1141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:19.482264 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:19.482264 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:19.482271 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:19.483709 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:19.484177 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:19.484178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:19.484179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:19.484233 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:19.484451 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:19.484452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:19.484452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:19.484595 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:19.484690 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:19.484828 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 23:07:19.484922 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":19}
I20260810 23:07:19.488376 14621 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd2248c5c18c46baaf3796872ecdb458: Generated new TSK 0
I20260810 23:07:19.488528 14621 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:19.492627 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 14534
2026-08-10T23:07:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1365 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:19 dist-test-slave-1xrh krb5kdc[14643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:19 dist-test-slave-1xrh krb5kdc[14643](info): set up 2 sockets
Aug 10 23:07:19 dist-test-slave-1xrh krb5kdc[14643](info): commencing operation
krb5kdc: starting...
W20260810 23:07:20.588388 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:20 dist-test-slave-1xrh krb5kdc[14643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:20.708223 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:20.789247 14656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:20.789505 14656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:20.789546 14656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:20.790714 14656 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:20.790798 14656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:20.790822 14656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:20.790844 14656 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:20.790874 14656 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:20.792439 14656 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:20.792724 14656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:20.792963 14656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:20.797849 14662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:20.797930 14664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:20.798009 14656 server_base.cc:1034] running on GCE node
W20260810 23:07:20.797845 14661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:20.798398 14656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:20.798672 14656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:20.799861 14656 hybrid_clock.cc:648] HybridClock initialized: now 1786403240799838 us; error 35 us; skew 500 ppm
Aug 10 23:07:20 dist-test-slave-1xrh krb5kdc[14643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:20.802316 14656 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:20.802735 14656 webserver.cc:460] Webserver started at http://127.0.0.1:44669/ using document root <none> and password file <none>
I20260810 23:07:20.802958 14656 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:20.803022 14656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:20.803196 14656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:20.804270 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "2f94ff9a6a154ce1acf696ebfe7124b0"
format_stamp: "Formatted at 2026-08-10 23:07:20 on dist-test-slave-1xrh"
server_key: "cec42dc996cfa05826ab988d6bc00fac"
server_key_iv: "df490c5c6e0a575f140d518249c3458b"
server_key_version: "encryptionkey@0"
I20260810 23:07:20.804632 14656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "2f94ff9a6a154ce1acf696ebfe7124b0"
format_stamp: "Formatted at 2026-08-10 23:07:20 on dist-test-slave-1xrh"
server_key: "cec42dc996cfa05826ab988d6bc00fac"
server_key_iv: "df490c5c6e0a575f140d518249c3458b"
server_key_version: "encryptionkey@0"
I20260810 23:07:20.805939 14656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:20.806632 14672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:20.806798 14656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:20.806893 14656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "2f94ff9a6a154ce1acf696ebfe7124b0"
format_stamp: "Formatted at 2026-08-10 23:07:20 on dist-test-slave-1xrh"
server_key: "cec42dc996cfa05826ab988d6bc00fac"
server_key_iv: "df490c5c6e0a575f140d518249c3458b"
server_key_version: "encryptionkey@0"
I20260810 23:07:20.806980 14656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:20.824559 14656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:20.826210 14656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:20.826360 14656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:20.830293 14656 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37181
I20260810 23:07:20.830329 14735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37181 every 8 connection(s)
I20260810 23:07:20.830678 14656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:20.831539 14737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:20.832813 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 14656
I20260810 23:07:20.832968 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:20.833150 23777 external_mini_cluster.cc:1428] Setting key e4ee07e3bce58a720c81b2a741ea2586
I20260810 23:07:20.834165 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0: Bootstrap starting.
I20260810 23:07:20.834863 14737 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:20.835198 14737 log.cc:824] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:20.835911 14737 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0: No bootstrap required, opened a new log
I20260810 23:07:20.838240 14737 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f94ff9a6a154ce1acf696ebfe7124b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } }
I20260810 23:07:20.838415 14737 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:20.838457 14737 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f94ff9a6a154ce1acf696ebfe7124b0, State: Initialized, Role: FOLLOWER
I20260810 23:07:20.838641 14737 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f94ff9a6a154ce1acf696ebfe7124b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } }
I20260810 23:07:20.838725 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:20.833690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53883 (local address 127.0.0.1:37181)
0810 23:07:20.833861 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:20.833869 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:20.833900 (+ 31us) server_negotiation.cc:407] Connection header received
0810 23:07:20.834321 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:20.834329 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:20.834393 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:20.834510 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:20.834776 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:20.835837 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:20.836918 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:20.837308 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:20.837387 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:20.838066 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:20.838070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:20.838070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:20.838224 (+ 154us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:20.838252 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:20.838262 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:20.838312 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:20.838437 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 23:07:20.838505 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":77,"threads_started":1}
I20260810 23:07:20.838730 14737 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:20.838814 14737 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:20.838865 14737 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:20.839567 14737 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f94ff9a6a154ce1acf696ebfe7124b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } }
I20260810 23:07:20.839656 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:20.839671 14737 leader_election.cc:302] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f94ff9a6a154ce1acf696ebfe7124b0; no voters:
I20260810 23:07:20.839831 14737 leader_election.cc:288] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:20.839854 14742 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:20.839969 14742 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [term 1 LEADER]: Becoming Leader. State: Replica: 2f94ff9a6a154ce1acf696ebfe7124b0, State: Running, Role: LEADER
I20260810 23:07:20.840085 14737 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:20.840111 14742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f94ff9a6a154ce1acf696ebfe7124b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } }
I20260810 23:07:20.840411 14742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f94ff9a6a154ce1acf696ebfe7124b0. Latest consensus state: current_term: 1 leader_uuid: "2f94ff9a6a154ce1acf696ebfe7124b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f94ff9a6a154ce1acf696ebfe7124b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } } }
I20260810 23:07:20.840505 14742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:20.840416 14743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f94ff9a6a154ce1acf696ebfe7124b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f94ff9a6a154ce1acf696ebfe7124b0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37181 } } }
I20260810 23:07:20.840652 14743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0 [sys.catalog]: This master's current role is: LEADER
W20260810 23:07:20.842182 14765 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:07:20.842283 14765 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:07:20.842406 14748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:20.842605 14748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:20.844187 14748 catalog_manager.cc:1347] Generated new cluster ID: 08f84961e80948ac9cde66865995da45
I20260810 23:07:20.844255 14748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:20.844578 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:20.841205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56041 (local address 127.0.0.1:37181)
0810 23:07:20.841238 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:20.841239 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:20.841363 (+ 124us) server_negotiation.cc:407] Connection header received
0810 23:07:20.841487 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:20.841488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:20.841499 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:20.841531 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:20.841746 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:20.842240 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:20.843209 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:20.843501 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:20.843571 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:20.844327 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:20.844328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:20.844329 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:20.844333 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:20.844347 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:20.844351 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:20.844425 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:20.844493 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 23:07:20.844513 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:07:20.849563 14748 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:20.850226 14748 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:20.853359 14748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f94ff9a6a154ce1acf696ebfe7124b0: Generated new TSK 0
I20260810 23:07:20.853519 14748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:20.855270 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 14656
2026-08-10T23:07:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1362 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:20 dist-test-slave-1xrh krb5kdc[14770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:20 dist-test-slave-1xrh krb5kdc[14770](info): set up 2 sockets
Aug 10 23:07:20 dist-test-slave-1xrh krb5kdc[14770](info): commencing operation
krb5kdc: starting...
W20260810 23:07:21.950467 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:22 dist-test-slave-1xrh krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:22.072201 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41015
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:22.154250 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:22.154461 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:22.154497 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:22.155669 14785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:22.155728 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:22.155752 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:22.155774 14785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:22.155812 14785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:22.157262 14785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:22.157522 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:22.157735 14785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:22.162266 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:22.162266 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:22.162290 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:22.162638 14785 server_base.cc:1034] running on GCE node
I20260810 23:07:22.162853 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:22.163159 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:22.164315 14785 hybrid_clock.cc:648] HybridClock initialized: now 1786403242164304 us; error 29 us; skew 500 ppm
Aug 10 23:07:22 dist-test-slave-1xrh krb5kdc[14770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:22.171749 14785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:22.172235 14785 webserver.cc:460] Webserver started at http://127.0.0.1:33847/ using document root <none> and password file <none>
I20260810 23:07:22.172492 14785 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:22.172539 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:22.172700 14785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:22.173748 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "5d7b9ea3007f45008b460e839de1caeb"
format_stamp: "Formatted at 2026-08-10 23:07:22 on dist-test-slave-1xrh"
server_key: "7d6db28b85877fe5a5696334d58f6af5"
server_key_iv: "9398809bf05b84b758d4344a5d05df0b"
server_key_version: "encryptionkey@0"
I20260810 23:07:22.174103 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "5d7b9ea3007f45008b460e839de1caeb"
format_stamp: "Formatted at 2026-08-10 23:07:22 on dist-test-slave-1xrh"
server_key: "7d6db28b85877fe5a5696334d58f6af5"
server_key_iv: "9398809bf05b84b758d4344a5d05df0b"
server_key_version: "encryptionkey@0"
I20260810 23:07:22.175563 14785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:22.176285 14801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:22.176481 14785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:22.176592 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "5d7b9ea3007f45008b460e839de1caeb"
format_stamp: "Formatted at 2026-08-10 23:07:22 on dist-test-slave-1xrh"
server_key: "7d6db28b85877fe5a5696334d58f6af5"
server_key_iv: "9398809bf05b84b758d4344a5d05df0b"
server_key_version: "encryptionkey@0"
I20260810 23:07:22.176700 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:22.179965 14785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:22.182152 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:22.182297 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:22.186125 14785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41015
I20260810 23:07:22.186161 14863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41015 every 8 connection(s)
I20260810 23:07:22.186481 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:22.186702 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 14785
I20260810 23:07:22.186843 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:22.187031 23777 external_mini_cluster.cc:1428] Setting key 574798a1afad55cf8f43491effa540df
I20260810 23:07:22.187642 14864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:22.189345 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:22.187504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56369 (local address 127.0.0.1:41015)
0810 23:07:22.187699 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:22.187709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:22.187743 (+ 34us) server_negotiation.cc:407] Connection header received
0810 23:07:22.188154 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:22.188160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:22.188184 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:22.188326 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:22.188947 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:22.188951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:22.188952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:22.188956 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:22.189001 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:22.189011 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:22.189084 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:22.189151 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 23:07:22.189191 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260810 23:07:22.190253 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:22.190639 14864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb: Bootstrap starting.
I20260810 23:07:22.191308 14864 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:22.191593 14864 log.cc:824] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:22.192404 14864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb: No bootstrap required, opened a new log
I20260810 23:07:22.192664 14867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:22.191694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39773 (local address 127.0.0.1:41015)
0810 23:07:22.191758 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:22.191760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:22.191823 (+ 63us) server_negotiation.cc:407] Connection header received
0810 23:07:22.191876 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:22.191877 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:22.191881 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:22.191934 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:22.192451 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:22.192453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:22.192454 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:22.192458 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:22.192473 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:22.192477 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:22.192527 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:22.192597 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 23:07:22.192609 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:07:22.194370 14864 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7b9ea3007f45008b460e839de1caeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41015 } }
I20260810 23:07:22.194509 14864 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:22.194531 14864 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d7b9ea3007f45008b460e839de1caeb, State: Initialized, Role: FOLLOWER
I20260810 23:07:22.194609 14864 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7b9ea3007f45008b460e839de1caeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41015 } }
I20260810 23:07:22.194666 14864 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:22.194685 14864 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:22.194716 14864 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:22.195283 14864 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7b9ea3007f45008b460e839de1caeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41015 } }
I20260810 23:07:22.195415 14864 leader_election.cc:302] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5d7b9ea3007f45008b460e839de1caeb; no voters:
I20260810 23:07:22.195592 14864 leader_election.cc:288] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:22.195640 14875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:22.195811 14875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [term 1 LEADER]: Becoming Leader. State: Replica: 5d7b9ea3007f45008b460e839de1caeb, State: Running, Role: LEADER
I20260810 23:07:22.195873 14864 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:22.195892 14875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7b9ea3007f45008b460e839de1caeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41015 } }
I20260810 23:07:22.196267 14876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5d7b9ea3007f45008b460e839de1caeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7b9ea3007f45008b460e839de1caeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41015 } } }
I20260810 23:07:22.196346 14876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:22.196306 14877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5d7b9ea3007f45008b460e839de1caeb. Latest consensus state: current_term: 1 leader_uuid: "5d7b9ea3007f45008b460e839de1caeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5d7b9ea3007f45008b460e839de1caeb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41015 } } }
I20260810 23:07:22.196461 14877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:22.196569 14881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:22.197078 14881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:22.198549 14881 catalog_manager.cc:1347] Generated new cluster ID: 2fff5983f96a4ece90c7d278a03c811c
I20260810 23:07:22.198599 14881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:22.225490 14881 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:22.225677 14881 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:22.231374 14881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5d7b9ea3007f45008b460e839de1caeb: Generated new TSK 0
I20260810 23:07:22.231542 14881 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:22.240532 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 14785
2026-08-10T23:07:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:22 dist-test-slave-1xrh krb5kdc[14898](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:22 dist-test-slave-1xrh krb5kdc[14898](info): set up 2 sockets
Aug 10 23:07:22 dist-test-slave-1xrh krb5kdc[14898](info): commencing operation
krb5kdc: starting...
W20260810 23:07:23.349270 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:23 dist-test-slave-1xrh krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:23.469758 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:23.550467 14913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:23.550668 14913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:23.550693 14913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:23.551797 14913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:23.551836 14913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:23.551849 14913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:23.551863 14913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:23.551875 14913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:23.553251 14913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:23.553455 14913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:23.553627 14913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:07:23.558598 14913 server_base.cc:1034] running on GCE node
W20260810 23:07:23.558518 14919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:23.558517 14921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:23.558544 14918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:23.559041 14913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:23.559345 14913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:23.560539 14913 hybrid_clock.cc:648] HybridClock initialized: now 1786403243560526 us; error 25 us; skew 500 ppm
Aug 10 23:07:23 dist-test-slave-1xrh krb5kdc[14898](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:23.563103 14913 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:23.563577 14913 webserver.cc:460] Webserver started at http://127.0.0.1:41357/ using document root <none> and password file <none>
I20260810 23:07:23.563818 14913 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:23.563885 14913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:23.564028 14913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:23.565032 14913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "501f13cb330b48aabe13787f726c7c29"
format_stamp: "Formatted at 2026-08-10 23:07:23 on dist-test-slave-1xrh"
server_key: "05f827d7fbe9f81cd36ea29e57d2df12"
server_key_iv: "9bc3fe7a23f0708086dfd667d94107f3"
server_key_version: "encryptionkey@0"
I20260810 23:07:23.565415 14913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "501f13cb330b48aabe13787f726c7c29"
format_stamp: "Formatted at 2026-08-10 23:07:23 on dist-test-slave-1xrh"
server_key: "05f827d7fbe9f81cd36ea29e57d2df12"
server_key_iv: "9bc3fe7a23f0708086dfd667d94107f3"
server_key_version: "encryptionkey@0"
I20260810 23:07:23.566701 14913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:23.567492 14929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:23.567684 14913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260810 23:07:23.567795 14913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "501f13cb330b48aabe13787f726c7c29"
format_stamp: "Formatted at 2026-08-10 23:07:23 on dist-test-slave-1xrh"
server_key: "05f827d7fbe9f81cd36ea29e57d2df12"
server_key_iv: "9bc3fe7a23f0708086dfd667d94107f3"
server_key_version: "encryptionkey@0"
I20260810 23:07:23.567868 14913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:23.583529 14913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:23.585711 14913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:23.585865 14913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:23.590137 14913 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41735
I20260810 23:07:23.590183 14987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41735 every 8 connection(s)
I20260810 23:07:23.590530 14913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:23.591393 14988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:23.593550 14988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29: Bootstrap starting.
I20260810 23:07:23.594144 14988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:23.594303 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 14913
I20260810 23:07:23.594429 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:23.594477 14988 log.cc:824] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:23.594635 23777 external_mini_cluster.cc:1428] Setting key 2fd20dfdd1c3d236f94488b47df8f538
I20260810 23:07:23.595309 14988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29: No bootstrap required, opened a new log
I20260810 23:07:23.597430 14988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f13cb330b48aabe13787f726c7c29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260810 23:07:23.597582 14988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:23.597645 14988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501f13cb330b48aabe13787f726c7c29, State: Initialized, Role: FOLLOWER
I20260810 23:07:23.597780 14988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f13cb330b48aabe13787f726c7c29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260810 23:07:23.597877 14988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:23.597919 14988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:23.597967 14988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:23.598520 14988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f13cb330b48aabe13787f726c7c29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260810 23:07:23.598635 14988 leader_election.cc:302] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 501f13cb330b48aabe13787f726c7c29; no voters:
I20260810 23:07:23.598793 14988 leader_election.cc:288] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:23.598830 14993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:23.598989 14993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [term 1 LEADER]: Becoming Leader. State: Replica: 501f13cb330b48aabe13787f726c7c29, State: Running, Role: LEADER
I20260810 23:07:23.599050 14988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:23.600533 14993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f13cb330b48aabe13787f726c7c29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } }
I20260810 23:07:23.601107 14994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 501f13cb330b48aabe13787f726c7c29. Latest consensus state: current_term: 1 leader_uuid: "501f13cb330b48aabe13787f726c7c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f13cb330b48aabe13787f726c7c29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } } }
I20260810 23:07:23.601154 14995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "501f13cb330b48aabe13787f726c7c29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "501f13cb330b48aabe13787f726c7c29" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41735 } } }
I20260810 23:07:23.601292 14994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:23.601300 14995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29 [sys.catalog]: This master's current role is: LEADER
Aug 10 23:07:23 dist-test-slave-1xrh krb5kdc[14898](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
W20260810 23:07:23.601752 15007 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:07:23.601822 15007 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:07:23.601929 15008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:23.602108 15008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:23.603729 15008 catalog_manager.cc:1347] Generated new cluster ID: 6b53fa11a69741889c05ccbee4cb0ccb
I20260810 23:07:23.603790 15008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:23.605691 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:23.595116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:58909 (local address 127.0.0.1:41735)
0810 23:07:23.595291 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:23.595302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:23.595337 (+ 35us) server_negotiation.cc:407] Connection header received
0810 23:07:23.595815 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:23.595822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:23.595871 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:23.596019 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:23.596642 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:23.597528 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:23.598411 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:23.598964 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:23.599471 (+ 507us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:23.601947 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:23.601951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:23.601951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:23.601964 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:23.604009 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:23.604454 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:23.604457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:23.604457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:23.604563 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:23.604788 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:23.604789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:23.604790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:23.605097 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:23.605272 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:23.605360 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 23:07:23.605476 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260810 23:07:23.606614 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:23.615097 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:23.608013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:38143 (local address 127.0.0.1:41735)
0810 23:07:23.608072 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:23.608074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:23.608291 (+ 217us) server_negotiation.cc:407] Connection header received
0810 23:07:23.608334 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:23.608335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:23.608345 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:23.608421 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:23.608974 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:23.609718 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:23.610687 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:23.611015 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:23.611082 (+ 67us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:23.612392 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:23.612393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:23.612393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:23.612401 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:23.613938 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:23.614324 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:23.614325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:23.614325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:23.614383 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:23.614595 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:23.614596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:23.614596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:23.614731 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:23.614830 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:23.614945 (+ 115us) server_negotiation.cc:299] Negotiation successful
0810 23:07:23.615009 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 23:07:23.630123 15008 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:23.630606 15008 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:23.636062 15008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 501f13cb330b48aabe13787f726c7c29: Generated new TSK 0
I20260810 23:07:23.636230 15008 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:23.640826 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 14913
W20260810 23:07:23.647955 14896 connection.cc:535] client connection to 127.0.0.1:41735 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41735): Connection reset by peer (error 104)
2026-08-10T23:07:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:23 dist-test-slave-1xrh krb5kdc[15020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:23 dist-test-slave-1xrh krb5kdc[15020](info): set up 2 sockets
Aug 10 23:07:23 dist-test-slave-1xrh krb5kdc[15020](info): commencing operation
krb5kdc: starting...
W20260810 23:07:24.735756 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:24 dist-test-slave-1xrh krb5kdc[15020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:24.858116 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44515
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44515
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:24.940276 15035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:24.940503 15035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:24.940555 15035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:24.941689 15035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:24.941751 15035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:24.941774 15035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:24.941811 15035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:24.941840 15035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:24.943254 15035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44515
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44515
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:24.943512 15035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:24.943723 15035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:24.948305 15043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:24.948329 15041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:24.948336 15040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:24.948493 15035 server_base.cc:1034] running on GCE node
I20260810 23:07:24.948751 15035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:24.949074 15035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:24.950237 15035 hybrid_clock.cc:648] HybridClock initialized: now 1786403244950228 us; error 37 us; skew 500 ppm
Aug 10 23:07:24 dist-test-slave-1xrh krb5kdc[15020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:24.952592 15035 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:24.953051 15035 webserver.cc:460] Webserver started at http://127.0.0.1:36183/ using document root <none> and password file <none>
I20260810 23:07:24.953258 15035 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:24.953317 15035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:24.953462 15035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:24.954463 15035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "6c1a97dc280443e5a99a283c106985dc"
format_stamp: "Formatted at 2026-08-10 23:07:24 on dist-test-slave-1xrh"
server_key: "0b7da7efd9427e5e7021e3312a840a33"
server_key_iv: "5efa660e7e9956260d56e4244c35ae20"
server_key_version: "encryptionkey@0"
I20260810 23:07:24.954825 15035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "6c1a97dc280443e5a99a283c106985dc"
format_stamp: "Formatted at 2026-08-10 23:07:24 on dist-test-slave-1xrh"
server_key: "0b7da7efd9427e5e7021e3312a840a33"
server_key_iv: "5efa660e7e9956260d56e4244c35ae20"
server_key_version: "encryptionkey@0"
I20260810 23:07:24.956195 15035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:24.956931 15052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:24.957094 15035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:24.957186 15035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "6c1a97dc280443e5a99a283c106985dc"
format_stamp: "Formatted at 2026-08-10 23:07:24 on dist-test-slave-1xrh"
server_key: "0b7da7efd9427e5e7021e3312a840a33"
server_key_iv: "5efa660e7e9956260d56e4244c35ae20"
server_key_version: "encryptionkey@0"
I20260810 23:07:24.957262 15035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:24.974864 15035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:24.977105 15035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:24.977267 15035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:24.981525 15035 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44515
I20260810 23:07:24.981573 15109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44515 every 8 connection(s)
I20260810 23:07:24.981935 15035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:24.982807 15111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:24.983520 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 15035
I20260810 23:07:24.983639 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:24.983861 23777 external_mini_cluster.cc:1428] Setting key 21578dc5f36854745a0bc91b00ae2019
I20260810 23:07:24.985678 15111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc: Bootstrap starting.
I20260810 23:07:24.986277 15111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:24.986555 15111 log.cc:824] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:24.987317 15111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc: No bootstrap required, opened a new log
I20260810 23:07:24.989164 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:24.984422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:36749 (local address 127.0.0.1:44515)
0810 23:07:24.984611 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:24.984621 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:24.984651 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:07:24.985061 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:24.985067 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:24.985113 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:24.985251 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:24.985435 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:24.986425 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:24.987251 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:24.987705 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:24.987819 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:24.988552 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:24.988556 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:24.988556 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:24.988756 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:24.988784 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:24.988794 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:24.988891 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:24.988968 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 23:07:24.988996 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":83,"threads_started":1}
I20260810 23:07:24.989498 15111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1a97dc280443e5a99a283c106985dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260810 23:07:24.989645 15111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:24.989683 15111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1a97dc280443e5a99a283c106985dc, State: Initialized, Role: FOLLOWER
I20260810 23:07:24.989820 15111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1a97dc280443e5a99a283c106985dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260810 23:07:24.989944 15111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:24.989988 15111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:24.990043 15111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:24.990097 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:24.990561 15111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1a97dc280443e5a99a283c106985dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260810 23:07:24.990650 15111 leader_election.cc:302] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c1a97dc280443e5a99a283c106985dc; no voters:
I20260810 23:07:24.990798 15111 leader_election.cc:288] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:24.990833 15117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:24.990949 15117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [term 1 LEADER]: Becoming Leader. State: Replica: 6c1a97dc280443e5a99a283c106985dc, State: Running, Role: LEADER
I20260810 23:07:24.991017 15117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1a97dc280443e5a99a283c106985dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } }
I20260810 23:07:24.991045 15111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:24.991377 15117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c1a97dc280443e5a99a283c106985dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1a97dc280443e5a99a283c106985dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } } }
I20260810 23:07:24.991532 15117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:24.991725 15117 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c1a97dc280443e5a99a283c106985dc. Latest consensus state: current_term: 1 leader_uuid: "6c1a97dc280443e5a99a283c106985dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c1a97dc280443e5a99a283c106985dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44515 } } }
I20260810 23:07:24.991876 15117 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:24.991789 15129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:24.993199 15129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:24.994670 15114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:24.991543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:41449 (local address 127.0.0.1:44515)
0810 23:07:24.991576 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:24.991578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:24.991586 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:24.991669 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:24.991669 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:24.991678 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:24.991719 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:24.991895 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:24.992554 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:24.993357 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:24.993554 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:24.993626 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:24.994418 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:24.994419 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:24.994419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:24.994424 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:24.994438 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:24.994442 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:24.994513 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:24.994590 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 23:07:24.994601 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:07:24.994900 15129 catalog_manager.cc:1347] Generated new cluster ID: cea2dbd3d562407eb6a6e47ed399ec33
I20260810 23:07:24.994951 15129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:25.017146 15129 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:25.017854 15129 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:25.031069 15129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c1a97dc280443e5a99a283c106985dc: Generated new TSK 0
I20260810 23:07:25.031270 15129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:25.039484 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 15035
W20260810 23:07:25.046335 15018 connection.cc:535] client connection to 127.0.0.1:44515 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44515): Connection reset by peer (error 104)
2026-08-10T23:07:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1398 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:25 dist-test-slave-1xrh krb5kdc[15145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:25 dist-test-slave-1xrh krb5kdc[15145](info): set up 2 sockets
Aug 10 23:07:25 dist-test-slave-1xrh krb5kdc[15145](info): commencing operation
krb5kdc: starting...
W20260810 23:07:26.134552 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:26 dist-test-slave-1xrh krb5kdc[15145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:26.256034 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37741
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37741
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:26.337828 15158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:26.338027 15158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:26.338050 15158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:26.339190 15158 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:26.339231 15158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:26.339246 15158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:26.339291 15158 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:26.339318 15158 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:26.340773 15158 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:26.341019 15158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:26.341231 15158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:26.345914 15166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:26.345961 15163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:26.345940 15164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:26.345961 15158 server_base.cc:1034] running on GCE node
I20260810 23:07:26.346374 15158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:26.346613 15158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:26.347744 15158 hybrid_clock.cc:648] HybridClock initialized: now 1786403246347732 us; error 28 us; skew 500 ppm
Aug 10 23:07:26 dist-test-slave-1xrh krb5kdc[15145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:26.350107 15158 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:26.350551 15158 webserver.cc:460] Webserver started at http://127.0.0.1:39837/ using document root <none> and password file <none>
I20260810 23:07:26.350773 15158 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:26.350818 15158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:26.350970 15158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:26.352057 15158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "1a2989a8b00d44f0ade6a2c0c6d94001"
format_stamp: "Formatted at 2026-08-10 23:07:26 on dist-test-slave-1xrh"
server_key: "1472f9f5f721614dcf2a228d8906f6c3"
server_key_iv: "a9418311389e4ba626b5b19e2848d6ea"
server_key_version: "encryptionkey@0"
I20260810 23:07:26.352398 15158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "1a2989a8b00d44f0ade6a2c0c6d94001"
format_stamp: "Formatted at 2026-08-10 23:07:26 on dist-test-slave-1xrh"
server_key: "1472f9f5f721614dcf2a228d8906f6c3"
server_key_iv: "a9418311389e4ba626b5b19e2848d6ea"
server_key_version: "encryptionkey@0"
I20260810 23:07:26.353705 15158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:07:26.354408 15174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:26.354590 15158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:26.354677 15158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "1a2989a8b00d44f0ade6a2c0c6d94001"
format_stamp: "Formatted at 2026-08-10 23:07:26 on dist-test-slave-1xrh"
server_key: "1472f9f5f721614dcf2a228d8906f6c3"
server_key_iv: "a9418311389e4ba626b5b19e2848d6ea"
server_key_version: "encryptionkey@0"
I20260810 23:07:26.354761 15158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:26.357939 15158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:26.359823 15158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:26.359970 15158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:26.363838 15158 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37741
I20260810 23:07:26.363873 15233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37741 every 8 connection(s)
I20260810 23:07:26.364205 15158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:26.365129 15234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:26.367314 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001: Bootstrap starting.
I20260810 23:07:26.367893 15234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:26.368175 15234 log.cc:824] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:26.368817 15234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001: No bootstrap required, opened a new log
I20260810 23:07:26.370343 15234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2989a8b00d44f0ade6a2c0c6d94001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37741 } }
I20260810 23:07:26.370492 15234 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:26.370548 15234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a2989a8b00d44f0ade6a2c0c6d94001, State: Initialized, Role: FOLLOWER
I20260810 23:07:26.370713 15234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2989a8b00d44f0ade6a2c0c6d94001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37741 } }
I20260810 23:07:26.370775 15234 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:26.370814 15234 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:26.370860 15234 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:26.370849 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 15158
I20260810 23:07:26.370976 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:26.371263 23777 external_mini_cluster.cc:1428] Setting key 3e58d3dfdd0b4b67e50008a7a32cdce9
I20260810 23:07:26.371605 15234 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2989a8b00d44f0ade6a2c0c6d94001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37741 } }
I20260810 23:07:26.371713 15234 leader_election.cc:302] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a2989a8b00d44f0ade6a2c0c6d94001; no voters:
I20260810 23:07:26.371891 15234 leader_election.cc:288] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:26.371929 15239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:26.372080 15239 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [term 1 LEADER]: Becoming Leader. State: Replica: 1a2989a8b00d44f0ade6a2c0c6d94001, State: Running, Role: LEADER
I20260810 23:07:26.372174 15234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:26.372238 15239 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2989a8b00d44f0ade6a2c0c6d94001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37741 } }
I20260810 23:07:26.372699 15240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a2989a8b00d44f0ade6a2c0c6d94001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2989a8b00d44f0ade6a2c0c6d94001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37741 } } }
I20260810 23:07:26.372797 15240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:26.372984 15241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a2989a8b00d44f0ade6a2c0c6d94001. Latest consensus state: current_term: 1 leader_uuid: "1a2989a8b00d44f0ade6a2c0c6d94001" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a2989a8b00d44f0ade6a2c0c6d94001" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37741 } } }
I20260810 23:07:26.373087 15241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:26.373358 15243 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:26.373811 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:26.371740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:50677 (local address 127.0.0.1:37741)
0810 23:07:26.371913 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:26.371923 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:26.372128 (+ 205us) server_negotiation.cc:407] Connection header received
0810 23:07:26.372255 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:26.372262 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:26.372290 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:26.372444 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:26.373265 (+ 821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:26.373269 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:26.373269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:26.373274 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:26.373330 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:26.373348 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:26.373396 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:26.373551 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:07:26.373603 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260810 23:07:26.373817 15243 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:26.375025 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:26.375757 15243 catalog_manager.cc:1347] Generated new cluster ID: fc9fafdb88ce4efeb01dfdd0398f6de2
I20260810 23:07:26.375813 15243 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:26.380601 15238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:26.379490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:56753 (local address 127.0.0.1:37741)
0810 23:07:26.379526 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:26.379528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:26.379708 (+ 180us) server_negotiation.cc:407] Connection header received
0810 23:07:26.379779 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:26.379780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:26.379786 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:26.379834 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:26.380382 (+ 548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:26.380383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:26.380384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:26.380388 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:26.380401 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:26.380404 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:26.380457 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:26.380511 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 23:07:26.380525 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:07:26.390568 15243 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:26.390740 15243 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:26.395643 15243 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a2989a8b00d44f0ade6a2c0c6d94001: Generated new TSK 0
I20260810 23:07:26.395804 15243 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:26.398452 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 15158
2026-08-10T23:07:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1361 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:26 dist-test-slave-1xrh krb5kdc[15266](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:26 dist-test-slave-1xrh krb5kdc[15266](info): set up 2 sockets
Aug 10 23:07:26 dist-test-slave-1xrh krb5kdc[15266](info): commencing operation
krb5kdc: starting...
W20260810 23:07:27.530480 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:27 dist-test-slave-1xrh krb5kdc[15266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:27.650631 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43647
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:27.741782 15281 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:27.742020 15281 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:27.742079 15281 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:27.743314 15281 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:27.743351 15281 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:27.743366 15281 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:27.743402 15281 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:27.743427 15281 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:27.744915 15281 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43647
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15281
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:27.745164 15281 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:27.745379 15281 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:27.750056 15286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:27.750264 15281 server_base.cc:1034] running on GCE node
W20260810 23:07:27.750062 15289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:27.750062 15287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:27.750597 15281 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:27.750872 15281 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:27.752028 15281 hybrid_clock.cc:648] HybridClock initialized: now 1786403247752016 us; error 41 us; skew 500 ppm
Aug 10 23:07:27 dist-test-slave-1xrh krb5kdc[15266](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:27.754513 15281 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:27.754917 15281 webserver.cc:460] Webserver started at http://127.0.0.1:34891/ using document root <none> and password file <none>
I20260810 23:07:27.755200 15281 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:27.755270 15281 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:27.755389 15281 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:27.756332 15281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "c3eebdf809a64598a48e0873cd9e45f7"
format_stamp: "Formatted at 2026-08-10 23:07:27 on dist-test-slave-1xrh"
server_key: "d2b4cf468a8d796da58691ed44242333"
server_key_iv: "333eafa538f35d93f474832a98926635"
server_key_version: "encryptionkey@0"
I20260810 23:07:27.756626 15281 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "c3eebdf809a64598a48e0873cd9e45f7"
format_stamp: "Formatted at 2026-08-10 23:07:27 on dist-test-slave-1xrh"
server_key: "d2b4cf468a8d796da58691ed44242333"
server_key_iv: "333eafa538f35d93f474832a98926635"
server_key_version: "encryptionkey@0"
I20260810 23:07:27.757833 15281 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 23:07:27.758473 15298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:27.758634 15281 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:27.758735 15281 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "c3eebdf809a64598a48e0873cd9e45f7"
format_stamp: "Formatted at 2026-08-10 23:07:27 on dist-test-slave-1xrh"
server_key: "d2b4cf468a8d796da58691ed44242333"
server_key_iv: "333eafa538f35d93f474832a98926635"
server_key_version: "encryptionkey@0"
I20260810 23:07:27.758805 15281 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:27.768374 15281 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:27.770547 15281 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:27.770705 15281 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:27.774534 15281 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43647
I20260810 23:07:27.774592 15359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43647 every 8 connection(s)
I20260810 23:07:27.774928 15281 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:27.775643 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 15281
I20260810 23:07:27.775765 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:27.775861 15360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:27.776000 23777 external_mini_cluster.cc:1428] Setting key f89ee56ca0a753478facbbc76e0e0919
I20260810 23:07:27.778903 15360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7: Bootstrap starting.
I20260810 23:07:27.779654 15360 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:27.779958 15360 log.cc:824] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:27.780642 15360 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7: No bootstrap required, opened a new log
I20260810 23:07:27.782593 15360 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eebdf809a64598a48e0873cd9e45f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } }
I20260810 23:07:27.782758 15360 raft_consensus.cc:374] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
Aug 10 23:07:27 dist-test-slave-1xrh krb5kdc[15266](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 23:07:27.782790 15360 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3eebdf809a64598a48e0873cd9e45f7, State: Initialized, Role: FOLLOWER
I20260810 23:07:27.782922 15360 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eebdf809a64598a48e0873cd9e45f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } }
I20260810 23:07:27.783022 15360 raft_consensus.cc:388] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:27.783066 15360 raft_consensus.cc:482] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:27.783113 15360 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:27.783890 15360 raft_consensus.cc:504] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eebdf809a64598a48e0873cd9e45f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } }
I20260810 23:07:27.784016 15360 leader_election.cc:302] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c3eebdf809a64598a48e0873cd9e45f7; no voters:
I20260810 23:07:27.784181 15360 leader_election.cc:288] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:27.784225 15365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:27.784375 15365 raft_consensus.cc:686] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [term 1 LEADER]: Becoming Leader. State: Replica: c3eebdf809a64598a48e0873cd9e45f7, State: Running, Role: LEADER
I20260810 23:07:27.784436 15360 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:27.784545 15365 consensus_queue.cc:237] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eebdf809a64598a48e0873cd9e45f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } }
I20260810 23:07:27.784924 15366 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c3eebdf809a64598a48e0873cd9e45f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eebdf809a64598a48e0873cd9e45f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } } }
I20260810 23:07:27.785018 15366 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:27.784920 15365 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c3eebdf809a64598a48e0873cd9e45f7. Latest consensus state: current_term: 1 leader_uuid: "c3eebdf809a64598a48e0873cd9e45f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c3eebdf809a64598a48e0873cd9e45f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43647 } } }
I20260810 23:07:27.785315 15365 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:27.785696 15372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:27.786306 15372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:27.787554 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:27.776484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:36657 (local address 127.0.0.1:43647)
0810 23:07:27.776678 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:27.776685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:27.776709 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:07:27.777277 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:27.777283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:27.777329 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:27.777448 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:27.778231 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:27.779162 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:27.780226 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:27.780678 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:27.780804 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:27.783368 (+ 2564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:27.783373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:27.783373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:27.783385 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:27.785613 (+ 2228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:27.786067 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:27.786071 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:27.786071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:27.786186 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:27.786464 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:27.786465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:27.786465 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:27.786777 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:27.786959 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:27.787245 (+ 286us) server_negotiation.cc:299] Negotiation successful
0810 23:07:27.787350 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":90,"threads_started":1}
I20260810 23:07:27.788314 15372 catalog_manager.cc:1347] Generated new cluster ID: 8eb0955ce92344e79133650f8fc0bd28
I20260810 23:07:27.788367 15372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:27.788618 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:27.799192 15363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:27.789981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:56879 (local address 127.0.0.1:43647)
0810 23:07:27.790017 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:27.790019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:27.791268 (+ 1249us) server_negotiation.cc:407] Connection header received
0810 23:07:27.791315 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:27.791316 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:27.791326 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:27.791398 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:27.791963 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:27.792703 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:27.793644 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:27.793980 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:27.794054 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:27.796339 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:27.796340 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:27.796341 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:27.796348 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:27.797987 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:27.798375 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:27.798376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:27.798376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:27.798444 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:27.798651 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:27.798652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:27.798653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:27.798776 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:27.798870 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:27.799003 (+ 133us) server_negotiation.cc:299] Negotiation successful
0810 23:07:27.799068 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:07:27.804375 15372 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:27.804833 15372 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:27.809074 15372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c3eebdf809a64598a48e0873cd9e45f7: Generated new TSK 0
I20260810 23:07:27.809257 15372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:27.815640 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 15281
W20260810 23:07:27.822744 15264 connection.cc:535] client connection to 127.0.0.1:43647 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43647): Connection reset by peer (error 104)
2026-08-10T23:07:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1414 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:27 dist-test-slave-1xrh krb5kdc[15393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:27 dist-test-slave-1xrh krb5kdc[15393](info): set up 2 sockets
Aug 10 23:07:27 dist-test-slave-1xrh krb5kdc[15393](info): commencing operation
krb5kdc: starting...
W20260810 23:07:28.909086 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:28 dist-test-slave-1xrh krb5kdc[15393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:29.027048 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33117
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44089
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:29.106750 15407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:29.106988 15407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:29.107028 15407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:29.108212 15407 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:29.108275 15407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:29.108301 15407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:29.108332 15407 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:29.108358 15407 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:29.109790 15407 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44089
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33117
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:29.110044 15407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:29.110252 15407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:29.115092 15415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:29.115166 15417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:29.115120 15414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:29.115201 15407 server_base.cc:1034] running on GCE node
I20260810 23:07:29.115530 15407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:29.115762 15407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:29.116915 15407 hybrid_clock.cc:648] HybridClock initialized: now 1786403249116893 us; error 37 us; skew 500 ppm
Aug 10 23:07:29 dist-test-slave-1xrh krb5kdc[15393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403249, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:29.119343 15407 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:29.119781 15407 webserver.cc:460] Webserver started at http://127.0.0.1:33451/ using document root <none> and password file <none>
I20260810 23:07:29.119990 15407 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:29.120052 15407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:29.120193 15407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:29.121188 15407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89"
format_stamp: "Formatted at 2026-08-10 23:07:29 on dist-test-slave-1xrh"
server_key: "daa4dadb87dd6d31f441952fcffe9df0"
server_key_iv: "44ccd2dc04c13d6692a613a87fec823a"
server_key_version: "encryptionkey@0"
I20260810 23:07:29.121536 15407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89"
format_stamp: "Formatted at 2026-08-10 23:07:29 on dist-test-slave-1xrh"
server_key: "daa4dadb87dd6d31f441952fcffe9df0"
server_key_iv: "44ccd2dc04c13d6692a613a87fec823a"
server_key_version: "encryptionkey@0"
I20260810 23:07:29.122838 15407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260810 23:07:29.123597 15425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:29.123759 15407 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:29.123883 15407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89"
format_stamp: "Formatted at 2026-08-10 23:07:29 on dist-test-slave-1xrh"
server_key: "daa4dadb87dd6d31f441952fcffe9df0"
server_key_iv: "44ccd2dc04c13d6692a613a87fec823a"
server_key_version: "encryptionkey@0"
I20260810 23:07:29.123973 15407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:29.135227 15407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:29.136909 15407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:29.137070 15407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:29.141198 15407 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33117
I20260810 23:07:29.141235 15484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33117 every 8 connection(s)
I20260810 23:07:29.141568 15407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:29.142334 15485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:29.144614 15485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89: Bootstrap starting.
I20260810 23:07:29.145169 15485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:29.145460 15485 log.cc:824] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:29.146136 15485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89: No bootstrap required, opened a new log
I20260810 23:07:29.147693 15485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260810 23:07:29.147846 15485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:29.147892 15485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f9062a2ea1c48f39a5dff7c94ed7e89, State: Initialized, Role: FOLLOWER
I20260810 23:07:29.148077 15485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260810 23:07:29.148173 15485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:29.148218 15485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:29.148272 15485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:29.148782 15485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260810 23:07:29.148906 15485 leader_election.cc:302] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f9062a2ea1c48f39a5dff7c94ed7e89; no voters:
I20260810 23:07:29.149085 15485 leader_election.cc:288] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:29.149137 15489 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:29.149331 15485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:29.149363 15489 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [term 1 LEADER]: Becoming Leader. State: Replica: 6f9062a2ea1c48f39a5dff7c94ed7e89, State: Running, Role: LEADER
I20260810 23:07:29.149448 15489 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } }
I20260810 23:07:29.149971 15491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f9062a2ea1c48f39a5dff7c94ed7e89. Latest consensus state: current_term: 1 leader_uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } } }
I20260810 23:07:29.150066 15491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:29.150297 15490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f9062a2ea1c48f39a5dff7c94ed7e89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33117 } } }
I20260810 23:07:29.150367 15500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:29.150454 15490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:29.150945 15500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:29.151584 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 15407
I20260810 23:07:29.151705 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:29.151913 23777 external_mini_cluster.cc:1428] Setting key f08ef0f1adf7471bde6bbf05e5d4b7da
I20260810 23:07:29.157156 15500 catalog_manager.cc:1347] Generated new cluster ID: 9fcbbc5fdfb84014a93ced5b6cd8e6cb
I20260810 23:07:29.157212 15500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:29.157778 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:29.152331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:60467 (local address 127.0.0.1:33117)
0810 23:07:29.152685 (+ 354us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:29.152694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:29.152724 (+ 30us) server_negotiation.cc:407] Connection header received
0810 23:07:29.152870 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:29.152877 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:29.152936 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:29.153074 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:29.153261 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:29.154651 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:29.155455 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:29.156134 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:29.156250 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:29.156918 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:29.156922 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:29.156922 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:29.157126 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:29.157167 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:29.157183 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:29.157265 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:29.157537 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 23:07:29.157565 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":62,"threads_started":1}
I20260810 23:07:29.158898 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:29.166682 15511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:29.160478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:59865 (local address 127.0.0.1:33117)
0810 23:07:29.160523 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:29.160524 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:29.163297 (+ 2773us) server_negotiation.cc:407] Connection header received
0810 23:07:29.163329 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:29.163330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:29.163341 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:29.163410 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:29.163529 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:29.164299 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:29.165237 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:29.165562 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:29.165631 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:29.166301 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:29.166302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:29.166303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:29.166307 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:29.166321 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:29.166326 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:29.166386 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:29.166600 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 23:07:29.166612 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:07:29.166986 15500 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:29.167512 15500 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:29.170797 15500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f9062a2ea1c48f39a5dff7c94ed7e89: Generated new TSK 0
I20260810 23:07:29.170949 15500 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:29.173979 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 15407
W20260810 23:07:29.182950 15391 connection.cc:535] client connection to 127.0.0.1:33117 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33117): Connection reset by peer (error 104)
2026-08-10T23:07:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:29 dist-test-slave-1xrh krb5kdc[15522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:29 dist-test-slave-1xrh krb5kdc[15522](info): set up 2 sockets
Aug 10 23:07:29 dist-test-slave-1xrh krb5kdc[15522](info): commencing operation
krb5kdc: starting...
W20260810 23:07:30.273839 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:30 dist-test-slave-1xrh krb5kdc[15522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 23:07:30.393317 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45351
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45351
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:30.473480 15537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:30.473662 15537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:30.473685 15537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:30.474821 15537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:30.474864 15537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:30.474877 15537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:30.474928 15537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:30.474951 15537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:30.476471 15537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45351
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:30.476703 15537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:30.476933 15537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:30.481729 15542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:30.481712 15543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:30.481715 15546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:30.481921 15537 server_base.cc:1034] running on GCE node
I20260810 23:07:30.482163 15537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:30.482443 15537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:30.483580 15537 hybrid_clock.cc:648] HybridClock initialized: now 1786403250483563 us; error 33 us; skew 500 ppm
Aug 10 23:07:30 dist-test-slave-1xrh krb5kdc[15522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:30.485973 15537 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 23:07:30.486398 15537 webserver.cc:460] Webserver started at http://127.0.0.1:41727/ using document root <none> and password file <none>
I20260810 23:07:30.486624 15537 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:30.486688 15537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:30.486829 15537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:30.487910 15537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "2509ebd72c1040cabf7319e25b1a809f"
format_stamp: "Formatted at 2026-08-10 23:07:30 on dist-test-slave-1xrh"
server_key: "0a5753b9ee267564d05f372c5b51d124"
server_key_iv: "26c786f9cd692b1d9efd977fe6d4e2d4"
server_key_version: "encryptionkey@0"
I20260810 23:07:30.488270 15537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "2509ebd72c1040cabf7319e25b1a809f"
format_stamp: "Formatted at 2026-08-10 23:07:30 on dist-test-slave-1xrh"
server_key: "0a5753b9ee267564d05f372c5b51d124"
server_key_iv: "26c786f9cd692b1d9efd977fe6d4e2d4"
server_key_version: "encryptionkey@0"
I20260810 23:07:30.489565 15537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 23:07:30.490275 15553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:30.490442 15537 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:30.490543 15537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "2509ebd72c1040cabf7319e25b1a809f"
format_stamp: "Formatted at 2026-08-10 23:07:30 on dist-test-slave-1xrh"
server_key: "0a5753b9ee267564d05f372c5b51d124"
server_key_iv: "26c786f9cd692b1d9efd977fe6d4e2d4"
server_key_version: "encryptionkey@0"
I20260810 23:07:30.490614 15537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:30.494014 15537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:30.496157 15537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:30.496304 15537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:30.500571 15537 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45351
I20260810 23:07:30.500623 15609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45351 every 8 connection(s)
I20260810 23:07:30.500931 15537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:30.501868 15610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:30.504130 15610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f: Bootstrap starting.
I20260810 23:07:30.504627 15610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:30.504875 15610 log.cc:824] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:30.505522 15610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f: No bootstrap required, opened a new log
I20260810 23:07:30.507045 15610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2509ebd72c1040cabf7319e25b1a809f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260810 23:07:30.507349 15610 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:30.507380 15610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2509ebd72c1040cabf7319e25b1a809f, State: Initialized, Role: FOLLOWER
I20260810 23:07:30.507508 15610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2509ebd72c1040cabf7319e25b1a809f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260810 23:07:30.507601 15610 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:30.507622 15610 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:30.507675 15610 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:30.507779 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 15537
I20260810 23:07:30.507902 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:30.508074 23777 external_mini_cluster.cc:1428] Setting key 207d7993c40c5f4efa751d06717bfb0e
I20260810 23:07:30.508330 15610 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2509ebd72c1040cabf7319e25b1a809f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260810 23:07:30.508451 15610 leader_election.cc:302] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2509ebd72c1040cabf7319e25b1a809f; no voters:
I20260810 23:07:30.508636 15610 leader_election.cc:288] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:30.508677 15615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:30.508864 15615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [term 1 LEADER]: Becoming Leader. State: Replica: 2509ebd72c1040cabf7319e25b1a809f, State: Running, Role: LEADER
I20260810 23:07:30.508958 15610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:30.508989 15615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2509ebd72c1040cabf7319e25b1a809f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } }
I20260810 23:07:30.509413 15615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2509ebd72c1040cabf7319e25b1a809f. Latest consensus state: current_term: 1 leader_uuid: "2509ebd72c1040cabf7319e25b1a809f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2509ebd72c1040cabf7319e25b1a809f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } } }
I20260810 23:07:30.509507 15615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:30.509680 15616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2509ebd72c1040cabf7319e25b1a809f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2509ebd72c1040cabf7319e25b1a809f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45351 } } }
I20260810 23:07:30.509812 15616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:30.510035 15619 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:30.510300 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:30.508497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:45965 (local address 127.0.0.1:45351)
0810 23:07:30.508677 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:30.508687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:30.508723 (+ 36us) server_negotiation.cc:407] Connection header received
0810 23:07:30.508862 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:30.508869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:30.508901 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:30.509292 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:30.509752 (+ 460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:30.509755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:30.509755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:30.509759 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:30.509799 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:30.509814 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:30.510038 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:30.510071 (+ 33us) server_negotiation.cc:299] Negotiation successful
0810 23:07:30.510125 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260810 23:07:30.510569 15619 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:30.511292 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:30.513680 15619 catalog_manager.cc:1347] Generated new cluster ID: aea1c9d0a5da4353b429ebac6c3a9b4a
I20260810 23:07:30.513731 15619 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:30.516220 15614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:30.512903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.186:34557 (local address 127.0.0.1:45351)
0810 23:07:30.512939 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:30.512941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:30.515306 (+ 2365us) server_negotiation.cc:407] Connection header received
0810 23:07:30.515365 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:30.515366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:30.515371 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:30.515444 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:30.516008 (+ 564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:30.516009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:30.516009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:30.516013 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:30.516028 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:30.516032 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:30.516095 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:30.516145 (+ 50us) server_negotiation.cc:299] Negotiation successful
0810 23:07:30.516158 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:07:30.522967 15619 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:30.523111 15619 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:30.527634 15619 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2509ebd72c1040cabf7319e25b1a809f: Generated new TSK 0
I20260810 23:07:30.527792 15619 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:30.530099 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 15537
2026-08-10T23:07:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:30 dist-test-slave-1xrh krb5kdc[15645](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:30 dist-test-slave-1xrh krb5kdc[15645](info): set up 2 sockets
Aug 10 23:07:30 dist-test-slave-1xrh krb5kdc[15645](info): commencing operation
krb5kdc: starting...
W20260810 23:07:31.617417 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:31 dist-test-slave-1xrh krb5kdc[15645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:31.734612 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:44059
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:44501
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:44059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:31.816977 15659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:31.817232 15659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:31.817274 15659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:31.818445 15659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:31.818514 15659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:31.818542 15659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:31.818575 15659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:31.818601 15659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:31.820217 15659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:44501
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:44059
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:44059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:31.820531 15659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:31.820770 15659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 23:07:31.825686 15659 server_base.cc:1034] running on GCE node
W20260810 23:07:31.825649 15664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:31.825655 15665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:31.825644 15667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:31.826009 15659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:31.826296 15659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:31.827457 15659 hybrid_clock.cc:648] HybridClock initialized: now 1786403251827440 us; error 47 us; skew 500 ppm
Aug 10 23:07:31 dist-test-slave-1xrh krb5kdc[15645](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403251, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:31.829878 15659 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:31.830349 15659 webserver.cc:460] Webserver started at http://127.23.56.126:32771/ using document root <none> and password file <none>
I20260810 23:07:31.830585 15659 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:31.830653 15659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:31.830801 15659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:31.831907 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "886340a4a55a43a49ca2edf1b070326a"
format_stamp: "Formatted at 2026-08-10 23:07:31 on dist-test-slave-1xrh"
server_key: "ff3d05dd7d8b88d72ce267c3815b1b3f"
server_key_iv: "941d96194adada6364a9a657463256f6"
server_key_version: "encryptionkey@0"
I20260810 23:07:31.832268 15659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "886340a4a55a43a49ca2edf1b070326a"
format_stamp: "Formatted at 2026-08-10 23:07:31 on dist-test-slave-1xrh"
server_key: "ff3d05dd7d8b88d72ce267c3815b1b3f"
server_key_iv: "941d96194adada6364a9a657463256f6"
server_key_version: "encryptionkey@0"
I20260810 23:07:31.833559 15659 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:07:31.834296 15675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:31.834462 15659 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:31.834564 15659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "886340a4a55a43a49ca2edf1b070326a"
format_stamp: "Formatted at 2026-08-10 23:07:31 on dist-test-slave-1xrh"
server_key: "ff3d05dd7d8b88d72ce267c3815b1b3f"
server_key_iv: "941d96194adada6364a9a657463256f6"
server_key_version: "encryptionkey@0"
I20260810 23:07:31.834636 15659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:31.846082 15659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:31.847977 15659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:31.848145 15659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:31.852303 15659 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:44059
I20260810 23:07:31.852341 15732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:44059 every 8 connection(s)
I20260810 23:07:31.852715 15659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:31.853575 15733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:31.855926 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a: Bootstrap starting.
I20260810 23:07:31.856495 15733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:31.856808 15733 log.cc:824] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:31.857508 15733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a: No bootstrap required, opened a new log
I20260810 23:07:31.859165 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 15659
I20260810 23:07:31.859285 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:31.859194 15733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886340a4a55a43a49ca2edf1b070326a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44059 } }
I20260810 23:07:31.859364 15733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:31.859411 15733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 886340a4a55a43a49ca2edf1b070326a, State: Initialized, Role: FOLLOWER
I20260810 23:07:31.859483 23777 external_mini_cluster.cc:1428] Setting key d5172ff757a1a2fd06c84de9ab713115
I20260810 23:07:31.859623 15733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886340a4a55a43a49ca2edf1b070326a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44059 } }
I20260810 23:07:31.859714 15733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:31.859736 15733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:31.859805 15733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:31.860713 15733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886340a4a55a43a49ca2edf1b070326a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44059 } }
I20260810 23:07:31.860829 15733 leader_election.cc:302] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 886340a4a55a43a49ca2edf1b070326a; no voters:
I20260810 23:07:31.860991 15733 leader_election.cc:288] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:31.861029 15738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:31.861212 15738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [term 1 LEADER]: Becoming Leader. State: Replica: 886340a4a55a43a49ca2edf1b070326a, State: Running, Role: LEADER
I20260810 23:07:31.861239 15733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:31.861366 15738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886340a4a55a43a49ca2edf1b070326a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44059 } }
I20260810 23:07:31.861828 15739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "886340a4a55a43a49ca2edf1b070326a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886340a4a55a43a49ca2edf1b070326a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44059 } } }
I20260810 23:07:31.861845 15740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 886340a4a55a43a49ca2edf1b070326a. Latest consensus state: current_term: 1 leader_uuid: "886340a4a55a43a49ca2edf1b070326a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "886340a4a55a43a49ca2edf1b070326a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44059 } } }
I20260810 23:07:31.862021 15739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:31.862035 15740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:31.862290 15749 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:31.862833 15749 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:31.864521 15749 catalog_manager.cc:1347] Generated new cluster ID: e44158d2c5c94f0e91268253ce408031
I20260810 23:07:31.864574 15749 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Aug 10 23:07:31 dist-test-slave-1xrh krb5kdc[15645](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:31.869982 15749 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:31.870589 15749 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:31.871373 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:31.860007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39873 (local address 127.23.56.126:44059)
0810 23:07:31.860185 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:31.860195 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:31.860237 (+ 42us) server_negotiation.cc:407] Connection header received
0810 23:07:31.860361 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:31.860366 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:31.860407 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:31.860537 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:31.861372 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:31.862747 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:31.863814 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:31.864412 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:31.864529 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:31.867598 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:31.867602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:31.867603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:31.867617 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:31.869556 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:31.870073 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:31.870077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:31.870077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:31.870200 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:31.870471 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:31.870472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:31.870472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:31.870746 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:31.870915 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:31.871034 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 23:07:31.871177 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260810 23:07:31.872355 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:31.878408 15749 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 886340a4a55a43a49ca2edf1b070326a: Generated new TSK 0
I20260810 23:07:31.878566 15749 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:31.880833 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:31.873823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54087 (local address 127.23.56.126:44059)
0810 23:07:31.873854 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:31.873855 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:31.873941 (+ 86us) server_negotiation.cc:407] Connection header received
0810 23:07:31.874006 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:31.874007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:31.874015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:31.874067 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:31.874670 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:31.875441 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:31.876432 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:31.876745 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:31.876808 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:31.878144 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:31.878145 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:31.878145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:31.878152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:31.879661 (+ 1509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:31.880020 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:31.880021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:31.880022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:31.880077 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:31.880281 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:31.880281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:31.880282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:31.880399 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:31.880492 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:31.880593 (+ 101us) server_negotiation.cc:299] Negotiation successful
0810 23:07:31.880655 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 23:07:31.882898 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 15659
W20260810 23:07:31.889493 15643 connection.cc:535] client connection to 127.23.56.126:44059 recv error: Network error: failed to read from TLS socket (remote: 127.23.56.126:44059): Connection reset by peer (error 104)
2026-08-10T23:07:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1351 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:31 dist-test-slave-1xrh krb5kdc[15769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:31 dist-test-slave-1xrh krb5kdc[15769](info): set up 2 sockets
Aug 10 23:07:31 dist-test-slave-1xrh krb5kdc[15769](info): commencing operation
krb5kdc: starting...
W20260810 23:07:32.970080 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:33 dist-test-slave-1xrh krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:33.090224 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:35345
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:40941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:35345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:33.173419 15782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:33.173609 15782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:33.173630 15782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:33.174664 15782 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:33.174697 15782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:33.174711 15782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:33.174723 15782 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:33.174734 15782 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:33.176120 15782 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:40941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:35345
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:35345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:33.176295 15782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:33.176453 15782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:33.181118 15789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:33.181082 15788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:33.181123 15782 server_base.cc:1034] running on GCE node
W20260810 23:07:33.181092 15791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:33.181592 15782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:33.181862 15782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:33.183020 15782 hybrid_clock.cc:648] HybridClock initialized: now 1786403253183001 us; error 37 us; skew 500 ppm
Aug 10 23:07:33 dist-test-slave-1xrh krb5kdc[15769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403253, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:33.191838 15782 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:33.192294 15782 webserver.cc:460] Webserver started at http://127.23.56.126:41689/ using document root <none> and password file <none>
I20260810 23:07:33.192516 15782 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:33.192560 15782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:33.192711 15782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:33.193727 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "c759fdda3ebf47268995148684b39e52"
format_stamp: "Formatted at 2026-08-10 23:07:33 on dist-test-slave-1xrh"
server_key: "b59702d821d0550c33c286a2bcb7f227"
server_key_iv: "4a4d976c51f7d2aad98003e2fb978275"
server_key_version: "encryptionkey@0"
I20260810 23:07:33.194072 15782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "c759fdda3ebf47268995148684b39e52"
format_stamp: "Formatted at 2026-08-10 23:07:33 on dist-test-slave-1xrh"
server_key: "b59702d821d0550c33c286a2bcb7f227"
server_key_iv: "4a4d976c51f7d2aad98003e2fb978275"
server_key_version: "encryptionkey@0"
I20260810 23:07:33.195427 15782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260810 23:07:33.196283 15798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:33.196504 15782 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:33.196584 15782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "c759fdda3ebf47268995148684b39e52"
format_stamp: "Formatted at 2026-08-10 23:07:33 on dist-test-slave-1xrh"
server_key: "b59702d821d0550c33c286a2bcb7f227"
server_key_iv: "4a4d976c51f7d2aad98003e2fb978275"
server_key_version: "encryptionkey@0"
I20260810 23:07:33.196677 15782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:33.220453 15782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:33.222419 15782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:33.222570 15782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:33.226795 15782 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:35345
I20260810 23:07:33.226858 15855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:35345 every 8 connection(s)
I20260810 23:07:33.227241 15782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:33.228083 15856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:33.230304 15856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52: Bootstrap starting.
I20260810 23:07:33.230882 15856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:33.231204 15856 log.cc:824] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:33.231874 15856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52: No bootstrap required, opened a new log
I20260810 23:07:33.233444 15856 raft_consensus.cc:348] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c759fdda3ebf47268995148684b39e52" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35345 } }
I20260810 23:07:33.233580 15856 raft_consensus.cc:374] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:33.233637 15856 raft_consensus.cc:729] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c759fdda3ebf47268995148684b39e52, State: Initialized, Role: FOLLOWER
I20260810 23:07:33.233784 15856 consensus_queue.cc:260] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c759fdda3ebf47268995148684b39e52" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35345 } }
I20260810 23:07:33.233845 15856 raft_consensus.cc:388] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:33.233908 15856 raft_consensus.cc:482] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:33.233979 15856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:33.234850 15856 raft_consensus.cc:504] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c759fdda3ebf47268995148684b39e52" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35345 } }
I20260810 23:07:33.234889 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 15782
I20260810 23:07:33.234998 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:33.234987 15856 leader_election.cc:302] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c759fdda3ebf47268995148684b39e52; no voters:
I20260810 23:07:33.235172 15856 leader_election.cc:288] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:33.235227 15859 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:33.235249 23777 external_mini_cluster.cc:1428] Setting key 9fbd28f20bfa7f2619e8ac88969dd80d
I20260810 23:07:33.235376 15859 raft_consensus.cc:686] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [term 1 LEADER]: Becoming Leader. State: Replica: c759fdda3ebf47268995148684b39e52, State: Running, Role: LEADER
I20260810 23:07:33.235433 15856 sys_catalog.cc:564] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:33.235503 15859 consensus_queue.cc:237] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c759fdda3ebf47268995148684b39e52" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35345 } }
I20260810 23:07:33.236092 15861 sys_catalog.cc:455] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c759fdda3ebf47268995148684b39e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c759fdda3ebf47268995148684b39e52" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35345 } } }
I20260810 23:07:33.236176 15861 sys_catalog.cc:458] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:33.236384 15860 sys_catalog.cc:455] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c759fdda3ebf47268995148684b39e52. Latest consensus state: current_term: 1 leader_uuid: "c759fdda3ebf47268995148684b39e52" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c759fdda3ebf47268995148684b39e52" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35345 } } }
I20260810 23:07:33.236526 15872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:33.236536 15860 sys_catalog.cc:458] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:33.237100 15872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:33.238597 15872 catalog_manager.cc:1347] Generated new cluster ID: fec5c8d44b2f41c488c1c8fc3920901f
I20260810 23:07:33.238653 15872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:33.241508 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:33.235683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47561 (local address 127.23.56.126:35345)
0810 23:07:33.235941 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:33.235951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:33.235994 (+ 43us) server_negotiation.cc:407] Connection header received
0810 23:07:33.236127 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:33.236134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:33.236208 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:33.236359 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:33.236757 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:33.238295 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:33.239433 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:33.240035 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:33.240150 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:33.240815 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:33.240820 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:33.240820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:33.241033 (+ 213us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:33.241073 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:33.241087 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:33.241170 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:33.241260 (+ 90us) server_negotiation.cc:299] Negotiation successful
0810 23:07:33.241288 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":65,"threads_started":1}
I20260810 23:07:33.242470 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:33.250837 15865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:33.247417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60515 (local address 127.23.56.126:35345)
0810 23:07:33.247454 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:33.247456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:33.247574 (+ 118us) server_negotiation.cc:407] Connection header received
0810 23:07:33.247635 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:33.247636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:33.247646 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:33.247696 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:33.247830 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:33.248551 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:33.249567 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:33.249871 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:33.249928 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:33.250597 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:33.250598 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:33.250598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:33.250603 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:33.250615 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:33.250619 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:33.250673 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:33.250756 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 23:07:33.250768 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:07:33.253214 15872 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:33.253705 15872 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:33.258589 15872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c759fdda3ebf47268995148684b39e52: Generated new TSK 0
I20260810 23:07:33.258764 15872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:33.261107 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 15782
W20260810 23:07:33.268913 15766 connection.cc:535] client connection to 127.23.56.126:35345 recv error: Network error: failed to read from TLS socket (remote: 127.23.56.126:35345): Connection reset by peer (error 104)
2026-08-10T23:07:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:33 dist-test-slave-1xrh krb5kdc[15889](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:33 dist-test-slave-1xrh krb5kdc[15889](info): set up 2 sockets
Aug 10 23:07:33 dist-test-slave-1xrh krb5kdc[15889](info): commencing operation
krb5kdc: starting...
W20260810 23:07:34.352715 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:34 dist-test-slave-1xrh krb5kdc[15889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:34.473845 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:42217
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:46283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:42217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:34.553933 15903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:34.554141 15903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:34.554163 15903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:34.555302 15903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:34.555342 15903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:34.555357 15903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:34.555370 15903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:34.555382 15903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:34.556934 15903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:46283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:42217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:42217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:34.557117 15903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:34.557279 15903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:34.562071 15914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:34.562049 15912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:34.562050 15903 server_base.cc:1034] running on GCE node
W20260810 23:07:34.562075 15911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:34.562531 15903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:34.562793 15903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:34.563934 15903 hybrid_clock.cc:648] HybridClock initialized: now 1786403254563922 us; error 49 us; skew 500 ppm
Aug 10 23:07:34 dist-test-slave-1xrh krb5kdc[15889](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403254, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:34.566270 15903 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:34.566715 15903 webserver.cc:460] Webserver started at http://127.23.56.126:40385/ using document root <none> and password file <none>
I20260810 23:07:34.566933 15903 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:34.567000 15903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:34.567194 15903 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:34.568208 15903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "917523f569e8443f94dc9c7d5183de9c"
format_stamp: "Formatted at 2026-08-10 23:07:34 on dist-test-slave-1xrh"
server_key: "8c1fb26ee5744a556019f5b22ab9f592"
server_key_iv: "866b26df187191b828ba1300f183d212"
server_key_version: "encryptionkey@0"
I20260810 23:07:34.568572 15903 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "917523f569e8443f94dc9c7d5183de9c"
format_stamp: "Formatted at 2026-08-10 23:07:34 on dist-test-slave-1xrh"
server_key: "8c1fb26ee5744a556019f5b22ab9f592"
server_key_iv: "866b26df187191b828ba1300f183d212"
server_key_version: "encryptionkey@0"
I20260810 23:07:34.569859 15903 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:07:34.570555 15921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:34.570729 15903 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:34.570827 15903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "917523f569e8443f94dc9c7d5183de9c"
format_stamp: "Formatted at 2026-08-10 23:07:34 on dist-test-slave-1xrh"
server_key: "8c1fb26ee5744a556019f5b22ab9f592"
server_key_iv: "866b26df187191b828ba1300f183d212"
server_key_version: "encryptionkey@0"
I20260810 23:07:34.570914 15903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:34.574126 15903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:34.575973 15903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:34.576128 15903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:34.580214 15903 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:42217
I20260810 23:07:34.580246 15977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:42217 every 8 connection(s)
I20260810 23:07:34.580577 15903 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:34.581519 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:34.583966 15978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c: Bootstrap starting.
I20260810 23:07:34.584543 15978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:34.584838 15978 log.cc:824] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:34.585522 15978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c: No bootstrap required, opened a new log
I20260810 23:07:34.587066 15978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917523f569e8443f94dc9c7d5183de9c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42217 } }
I20260810 23:07:34.587270 15978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:34.587327 15978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 917523f569e8443f94dc9c7d5183de9c, State: Initialized, Role: FOLLOWER
I20260810 23:07:34.587495 15978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917523f569e8443f94dc9c7d5183de9c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42217 } }
I20260810 23:07:34.587576 15978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:34.587616 15978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:34.587664 15978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:34.588042 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 15903
I20260810 23:07:34.588132 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:34.588315 23777 external_mini_cluster.cc:1428] Setting key a6359844cf5e607f4a33df980093dfb8
I20260810 23:07:34.588672 15978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917523f569e8443f94dc9c7d5183de9c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42217 } }
I20260810 23:07:34.588795 15978 leader_election.cc:302] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 917523f569e8443f94dc9c7d5183de9c; no voters:
I20260810 23:07:34.589001 15978 leader_election.cc:288] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:34.589041 15983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:34.589226 15983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [term 1 LEADER]: Becoming Leader. State: Replica: 917523f569e8443f94dc9c7d5183de9c, State: Running, Role: LEADER
I20260810 23:07:34.589252 15978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:34.589378 15983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917523f569e8443f94dc9c7d5183de9c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42217 } }
I20260810 23:07:34.590360 15986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "917523f569e8443f94dc9c7d5183de9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917523f569e8443f94dc9c7d5183de9c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42217 } } }
I20260810 23:07:34.590510 15986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:34.590652 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:34.588777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36641 (local address 127.23.56.126:42217)
0810 23:07:34.588982 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:34.588992 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:34.589049 (+ 57us) server_negotiation.cc:407] Connection header received
0810 23:07:34.589173 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:34.589180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:34.589209 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:34.589358 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:34.590002 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:34.590005 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:34.590006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:34.590010 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:34.590061 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:34.590075 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:34.590136 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:34.590400 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 23:07:34.590449 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260810 23:07:34.590917 15993 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:34.591521 15993 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:34.590224 15985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 917523f569e8443f94dc9c7d5183de9c. Latest consensus state: current_term: 1 leader_uuid: "917523f569e8443f94dc9c7d5183de9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "917523f569e8443f94dc9c7d5183de9c" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 42217 } } }
I20260810 23:07:34.591813 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:34.593144 15993 catalog_manager.cc:1347] Generated new cluster ID: d3c5525c783e40319381dadc71985dc3
I20260810 23:07:34.593240 15993 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:34.594362 15982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:34.593337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37781 (local address 127.23.56.126:42217)
0810 23:07:34.593368 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:34.593369 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:34.593505 (+ 136us) server_negotiation.cc:407] Connection header received
0810 23:07:34.593553 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:34.593554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:34.593559 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:34.593612 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:34.594168 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:34.594169 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:34.594169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:34.594173 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:34.594186 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:34.594190 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:34.594240 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:34.594294 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 23:07:34.594305 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:07:34.591794 15985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:34.614179 15993 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:34.614360 15993 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:34.626788 15993 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 917523f569e8443f94dc9c7d5183de9c: Generated new TSK 0
I20260810 23:07:34.626991 15993 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:34.639592 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 15903
2026-08-10T23:07:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1380 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:34 dist-test-slave-1xrh krb5kdc[16010](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:34 dist-test-slave-1xrh krb5kdc[16010](info): set up 2 sockets
Aug 10 23:07:34 dist-test-slave-1xrh krb5kdc[16010](info): commencing operation
krb5kdc: starting...
W20260810 23:07:35.728912 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:35 dist-test-slave-1xrh krb5kdc[16010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:35.850060 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:44477
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:43969
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:44477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:35.931207 16024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:35.931392 16024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:35.931413 16024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:35.932547 16024 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:35.932588 16024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:35.932601 16024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:35.932650 16024 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:35.932672 16024 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:35.934171 16024 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:43969
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:44477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:35.934402 16024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:35.934621 16024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:35.939249 16030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:35.939229 16032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:35.939239 16029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:35.939563 16024 server_base.cc:1034] running on GCE node
I20260810 23:07:35.939807 16024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:35.940075 16024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:35.941233 16024 hybrid_clock.cc:648] HybridClock initialized: now 1786403255941228 us; error 35 us; skew 500 ppm
Aug 10 23:07:35 dist-test-slave-1xrh krb5kdc[16010](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403255, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:35.943797 16024 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:35.944262 16024 webserver.cc:460] Webserver started at http://127.23.56.126:39433/ using document root <none> and password file <none>
I20260810 23:07:35.944478 16024 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:35.944540 16024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:35.944687 16024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:35.945720 16024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "2f5b3b309a3140c59b377b4c98c1d23a"
format_stamp: "Formatted at 2026-08-10 23:07:35 on dist-test-slave-1xrh"
server_key: "0f192c558c9c3b6f52d4547ddee2326a"
server_key_iv: "7cf7374882bf01fc678236d1da7f7e47"
server_key_version: "encryptionkey@0"
I20260810 23:07:35.946089 16024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "2f5b3b309a3140c59b377b4c98c1d23a"
format_stamp: "Formatted at 2026-08-10 23:07:35 on dist-test-slave-1xrh"
server_key: "0f192c558c9c3b6f52d4547ddee2326a"
server_key_iv: "7cf7374882bf01fc678236d1da7f7e47"
server_key_version: "encryptionkey@0"
I20260810 23:07:35.947386 16024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:07:35.948079 16041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:35.948249 16024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:35.948348 16024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "2f5b3b309a3140c59b377b4c98c1d23a"
format_stamp: "Formatted at 2026-08-10 23:07:35 on dist-test-slave-1xrh"
server_key: "0f192c558c9c3b6f52d4547ddee2326a"
server_key_iv: "7cf7374882bf01fc678236d1da7f7e47"
server_key_version: "encryptionkey@0"
I20260810 23:07:35.948418 16024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:35.976811 16024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:35.979115 16024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:35.979306 16024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:35.983065 16024 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:44477
I20260810 23:07:35.983101 16100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:44477 every 8 connection(s)
I20260810 23:07:35.983490 16024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:35.984335 16101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:35.984583 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 16024
I20260810 23:07:35.984711 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:35.984890 23777 external_mini_cluster.cc:1428] Setting key 2533067fa6b6114578fe7e57f4c81840
I20260810 23:07:35.987401 16101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a: Bootstrap starting.
I20260810 23:07:35.987999 16101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:35.988334 16101 log.cc:824] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:35.989082 16101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a: No bootstrap required, opened a new log
Aug 10 23:07:35 dist-test-slave-1xrh krb5kdc[16010](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:35.991603 16101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5b3b309a3140c59b377b4c98c1d23a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44477 } }
I20260810 23:07:35.991765 16101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:35.991811 16101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f5b3b309a3140c59b377b4c98c1d23a, State: Initialized, Role: FOLLOWER
I20260810 23:07:35.991940 16101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5b3b309a3140c59b377b4c98c1d23a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44477 } }
I20260810 23:07:35.992023 16101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:35.992054 16101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:35.992132 16101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:35.992838 16101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5b3b309a3140c59b377b4c98c1d23a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44477 } }
I20260810 23:07:35.992981 16101 leader_election.cc:302] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f5b3b309a3140c59b377b4c98c1d23a; no voters:
I20260810 23:07:35.993177 16101 leader_election.cc:288] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:35.993233 16106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:35.993381 16106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [term 1 LEADER]: Becoming Leader. State: Replica: 2f5b3b309a3140c59b377b4c98c1d23a, State: Running, Role: LEADER
I20260810 23:07:35.993553 16106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5b3b309a3140c59b377b4c98c1d23a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44477 } }
I20260810 23:07:35.993423 16101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:35.993961 16109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f5b3b309a3140c59b377b4c98c1d23a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5b3b309a3140c59b377b4c98c1d23a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44477 } } }
I20260810 23:07:35.993970 16108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f5b3b309a3140c59b377b4c98c1d23a. Latest consensus state: current_term: 1 leader_uuid: "2f5b3b309a3140c59b377b4c98c1d23a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f5b3b309a3140c59b377b4c98c1d23a" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 44477 } } }
I20260810 23:07:35.994138 16108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:35.994350 16109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:35.994714 16111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:35.995272 16111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:35.996248 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:35.985415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39917 (local address 127.23.56.126:44477)
0810 23:07:35.985601 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:35.985609 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:35.985633 (+ 24us) server_negotiation.cc:407] Connection header received
0810 23:07:35.986055 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:35.986061 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:35.986103 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:35.986215 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:35.987029 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:35.987987 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:35.988997 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:35.989427 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:35.989523 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:35.992032 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:35.992036 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:35.992037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:35.992048 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:35.993933 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:35.994395 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:35.994398 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:35.994398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:35.994512 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:35.994799 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:35.994801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:35.994801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:35.995173 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:35.995824 (+ 651us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:35.995912 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 23:07:35.996060 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260810 23:07:35.996932 16111 catalog_manager.cc:1347] Generated new cluster ID: a08f861cede14102909163f2b482fbd4
I20260810 23:07:35.996978 16111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:35.997229 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:36.009727 16104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:36.003394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54517 (local address 127.23.56.126:44477)
0810 23:07:36.003431 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:36.003433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:36.003573 (+ 140us) server_negotiation.cc:407] Connection header received
0810 23:07:36.003645 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:36.003646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:36.003656 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:36.003710 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:36.004241 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:36.004705 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:36.005566 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:36.005779 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:36.005862 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:36.007059 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:36.007060 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:36.007060 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:36.007067 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:36.008581 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:36.008950 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:36.008951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:36.008951 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:36.009011 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:36.009210 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:36.009211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:36.009212 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:36.009321 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:36.009432 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:36.009538 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 23:07:36.009621 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:07:36.032279 16111 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:36.032814 16111 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:36.043110 16111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f5b3b309a3140c59b377b4c98c1d23a: Generated new TSK 0
I20260810 23:07:36.043332 16111 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:36.049861 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 16024
2026-08-10T23:07:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:36 dist-test-slave-1xrh krb5kdc[16135](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:36 dist-test-slave-1xrh krb5kdc[16135](info): set up 2 sockets
Aug 10 23:07:36 dist-test-slave-1xrh krb5kdc[16135](info): commencing operation
krb5kdc: starting...
W20260810 23:07:37.143667 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:37 dist-test-slave-1xrh krb5kdc[16135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:37.263655 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33195
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:36905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:33195
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:37.345857 16151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:37.346040 16151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:37.346062 16151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:37.347182 16151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:37.347222 16151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:37.347235 16151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:37.347249 16151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:37.347297 16151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:37.348727 16151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:36905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:33195
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:33195
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:37.348963 16151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:37.349179 16151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:37.353921 16158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:37.353965 16151 server_base.cc:1034] running on GCE node
W20260810 23:07:37.353946 16157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:37.353917 16160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:37.354359 16151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:37.354647 16151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:37.355801 16151 hybrid_clock.cc:648] HybridClock initialized: now 1786403257355785 us; error 45 us; skew 500 ppm
Aug 10 23:07:37 dist-test-slave-1xrh krb5kdc[16135](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403257, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:37.358206 16151 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:37.358633 16151 webserver.cc:460] Webserver started at http://127.23.56.126:34797/ using document root <none> and password file <none>
I20260810 23:07:37.358840 16151 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:37.358912 16151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:37.359053 16151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:37.360049 16151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "0976ad1046e643048d2fc0797537ebb2"
format_stamp: "Formatted at 2026-08-10 23:07:37 on dist-test-slave-1xrh"
server_key: "c308c1ca77af6f842d20dcbac698525f"
server_key_iv: "7704196a578a8f8600fa64b9ae70db02"
server_key_version: "encryptionkey@0"
I20260810 23:07:37.360387 16151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "0976ad1046e643048d2fc0797537ebb2"
format_stamp: "Formatted at 2026-08-10 23:07:37 on dist-test-slave-1xrh"
server_key: "c308c1ca77af6f842d20dcbac698525f"
server_key_iv: "7704196a578a8f8600fa64b9ae70db02"
server_key_version: "encryptionkey@0"
I20260810 23:07:37.361634 16151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260810 23:07:37.362366 16167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:37.362548 16151 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:37.362641 16151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "0976ad1046e643048d2fc0797537ebb2"
format_stamp: "Formatted at 2026-08-10 23:07:37 on dist-test-slave-1xrh"
server_key: "c308c1ca77af6f842d20dcbac698525f"
server_key_iv: "7704196a578a8f8600fa64b9ae70db02"
server_key_version: "encryptionkey@0"
I20260810 23:07:37.362715 16151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:37.371850 16151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:37.373603 16151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:37.373751 16151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:37.377826 16151 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:33195
I20260810 23:07:37.377874 16226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:33195 every 8 connection(s)
I20260810 23:07:37.378218 16151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:37.379010 16227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:37.381157 16227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2: Bootstrap starting.
I20260810 23:07:37.381727 16227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:37.382020 16227 log.cc:824] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:37.382647 16227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2: No bootstrap required, opened a new log
I20260810 23:07:37.384264 16227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976ad1046e643048d2fc0797537ebb2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33195 } }
I20260810 23:07:37.384392 16227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:37.384414 16227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0976ad1046e643048d2fc0797537ebb2, State: Initialized, Role: FOLLOWER
I20260810 23:07:37.384588 16227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976ad1046e643048d2fc0797537ebb2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33195 } }
I20260810 23:07:37.384651 16227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:37.384672 16227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:37.384739 16227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:37.385522 16227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976ad1046e643048d2fc0797537ebb2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33195 } }
I20260810 23:07:37.385627 16227 leader_election.cc:302] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0976ad1046e643048d2fc0797537ebb2; no voters:
I20260810 23:07:37.385794 16227 leader_election.cc:288] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:37.385830 16230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:37.386139 16230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [term 1 LEADER]: Becoming Leader. State: Replica: 0976ad1046e643048d2fc0797537ebb2, State: Running, Role: LEADER
I20260810 23:07:37.386252 16230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976ad1046e643048d2fc0797537ebb2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33195 } }
I20260810 23:07:37.386291 16227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:37.386729 16231 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0976ad1046e643048d2fc0797537ebb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976ad1046e643048d2fc0797537ebb2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33195 } } }
I20260810 23:07:37.386866 16231 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:37.386749 16232 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0976ad1046e643048d2fc0797537ebb2. Latest consensus state: current_term: 1 leader_uuid: "0976ad1046e643048d2fc0797537ebb2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0976ad1046e643048d2fc0797537ebb2" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 33195 } } }
I20260810 23:07:37.387050 16232 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:37.387380 16240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:37.387866 16240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:37.388113 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 16151
I20260810 23:07:37.388228 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:37.388417 23777 external_mini_cluster.cc:1428] Setting key e922ebe05d8545ae070af690ecb27875
I20260810 23:07:37.389531 16240 catalog_manager.cc:1347] Generated new cluster ID: 4e5bf7e7014d44b18f280e201bdcc502
I20260810 23:07:37.389583 16240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:37.396469 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:37.388838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57251 (local address 127.23.56.126:33195)
0810 23:07:37.389010 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:37.389019 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:37.391357 (+ 2338us) server_negotiation.cc:407] Connection header received
0810 23:07:37.391487 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:37.391494 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:37.391556 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:37.391683 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:37.391914 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:37.393221 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:37.394373 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:37.394977 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:37.395079 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:37.395809 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:37.395813 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:37.395813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:37.396013 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:37.396047 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:37.396060 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:37.396117 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:37.396214 (+ 97us) server_negotiation.cc:299] Negotiation successful
0810 23:07:37.396269 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
I20260810 23:07:37.397375 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:37.400327 16240 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:37.400795 16240 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:37.406236 16240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0976ad1046e643048d2fc0797537ebb2: Generated new TSK 0
I20260810 23:07:37.406391 16240 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:37.407476 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:37.398794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38737 (local address 127.23.56.126:33195)
0810 23:07:37.398825 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:37.398827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:37.399310 (+ 483us) server_negotiation.cc:407] Connection header received
0810 23:07:37.399367 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:37.399367 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:37.399377 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:37.399439 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:37.399571 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:37.400157 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:37.406266 (+ 6109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:37.406587 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:37.406672 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:37.407248 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:37.407249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:37.407249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:37.407255 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:37.407268 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:37.407272 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:37.407332 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:37.407370 (+ 38us) server_negotiation.cc:299] Negotiation successful
0810 23:07:37.407404 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260810 23:07:37.409281 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 16151
2026-08-10T23:07:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:37 dist-test-slave-1xrh krb5kdc[16263](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:37 dist-test-slave-1xrh krb5kdc[16263](info): set up 2 sockets
Aug 10 23:07:37 dist-test-slave-1xrh krb5kdc[16263](info): commencing operation
krb5kdc: starting...
W20260810 23:07:38.504313 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:38 dist-test-slave-1xrh krb5kdc[16263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:38.623265 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:35347
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:41145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:35347
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:38.704460 16277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:38.704671 16277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:38.704694 16277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:38.705845 16277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:38.705890 16277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:38.705911 16277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:38.705924 16277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:38.705937 16277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:38.707468 16277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:41145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:35347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:35347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:38.707719 16277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:38.707952 16277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:38.712780 16284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:38.712877 16277 server_base.cc:1034] running on GCE node
W20260810 23:07:38.712764 16283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:38.712755 16286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:38.713207 16277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:38.713449 16277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:38.714586 16277 hybrid_clock.cc:648] HybridClock initialized: now 1786403258714584 us; error 36 us; skew 500 ppm
Aug 10 23:07:38 dist-test-slave-1xrh krb5kdc[16263](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403258, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:38.722043 16277 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:38.722554 16277 webserver.cc:460] Webserver started at http://127.23.56.126:35735/ using document root <none> and password file <none>
I20260810 23:07:38.722788 16277 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:38.722857 16277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:38.723016 16277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:38.724128 16277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "cff7932ff3f1478d906e7efc868c2223"
format_stamp: "Formatted at 2026-08-10 23:07:38 on dist-test-slave-1xrh"
server_key: "8c48729fc212adcb95f8d8872a7802cb"
server_key_iv: "b63e69782afce82ee4bff543c0b1d650"
server_key_version: "encryptionkey@0"
I20260810 23:07:38.724505 16277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "cff7932ff3f1478d906e7efc868c2223"
format_stamp: "Formatted at 2026-08-10 23:07:38 on dist-test-slave-1xrh"
server_key: "8c48729fc212adcb95f8d8872a7802cb"
server_key_iv: "b63e69782afce82ee4bff543c0b1d650"
server_key_version: "encryptionkey@0"
I20260810 23:07:38.725847 16277 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 23:07:38.726567 16293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:38.726738 16277 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:38.726840 16277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "cff7932ff3f1478d906e7efc868c2223"
format_stamp: "Formatted at 2026-08-10 23:07:38 on dist-test-slave-1xrh"
server_key: "8c48729fc212adcb95f8d8872a7802cb"
server_key_iv: "b63e69782afce82ee4bff543c0b1d650"
server_key_version: "encryptionkey@0"
I20260810 23:07:38.726922 16277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:38.730202 16277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:38.732486 16277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:38.732640 16277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:38.736763 16277 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:35347
I20260810 23:07:38.736809 16353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:35347 every 8 connection(s)
I20260810 23:07:38.737146 16277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:38.737406 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 16277
I20260810 23:07:38.737533 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:38.737725 23777 external_mini_cluster.cc:1428] Setting key a66258b5e83887e1bfd2f2ad005228e1
I20260810 23:07:38.738377 16354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:38.740357 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:38.738209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40061 (local address 127.23.56.126:35347)
0810 23:07:38.738440 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:38.738449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:38.738482 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:07:38.739045 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:38.739054 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:38.739088 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:38.739269 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:38.739866 (+ 597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:38.739869 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:38.739869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:38.739874 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:38.739921 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:38.739936 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:38.740021 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:38.740098 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 23:07:38.740141 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":101,"threads_started":1}
I20260810 23:07:38.741374 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:38.741746 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223: Bootstrap starting.
I20260810 23:07:38.742377 16354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:38.742690 16354 log.cc:824] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:38.743386 16354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223: No bootstrap required, opened a new log
I20260810 23:07:38.743952 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:38.742801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54219 (local address 127.23.56.126:35347)
0810 23:07:38.742849 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:38.742851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:38.742944 (+ 93us) server_negotiation.cc:407] Connection header received
0810 23:07:38.743014 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:38.743015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:38.743020 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:38.743069 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:38.743726 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:38.743727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:38.743727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 23:07:38.743732 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:38.743744 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 23:07:38.743748 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:38.743817 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:38.743869 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 23:07:38.743883 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:07:38.745425 16354 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff7932ff3f1478d906e7efc868c2223" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35347 } }
I20260810 23:07:38.745553 16354 raft_consensus.cc:374] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 23:07:38.745577 16354 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff7932ff3f1478d906e7efc868c2223, State: Initialized, Role: FOLLOWER
I20260810 23:07:38.745720 16354 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff7932ff3f1478d906e7efc868c2223" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35347 } }
I20260810 23:07:38.745793 16354 raft_consensus.cc:388] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 23:07:38.745814 16354 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 23:07:38.745884 16354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:38.746423 16354 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff7932ff3f1478d906e7efc868c2223" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35347 } }
I20260810 23:07:38.746534 16354 leader_election.cc:302] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff7932ff3f1478d906e7efc868c2223; no voters:
I20260810 23:07:38.746716 16354 leader_election.cc:288] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 23:07:38.746742 16365 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:38.746898 16365 raft_consensus.cc:686] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [term 1 LEADER]: Becoming Leader. State: Replica: cff7932ff3f1478d906e7efc868c2223, State: Running, Role: LEADER
I20260810 23:07:38.746973 16354 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:38.747032 16365 consensus_queue.cc:237] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff7932ff3f1478d906e7efc868c2223" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35347 } }
I20260810 23:07:38.747417 16367 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff7932ff3f1478d906e7efc868c2223. Latest consensus state: current_term: 1 leader_uuid: "cff7932ff3f1478d906e7efc868c2223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff7932ff3f1478d906e7efc868c2223" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35347 } } }
I20260810 23:07:38.747656 16367 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:38.747414 16366 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cff7932ff3f1478d906e7efc868c2223" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff7932ff3f1478d906e7efc868c2223" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 35347 } } }
I20260810 23:07:38.748067 16374 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:38.748088 16366 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223 [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:38.748610 16374 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:38.750022 16374 catalog_manager.cc:1347] Generated new cluster ID: 5d49276b59a24c1dbcc4eaf134fc2e92
I20260810 23:07:38.750073 16374 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:38.759878 16374 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:38.760051 16374 catalog_manager.cc:1504] Loading token signing keys...
I20260810 23:07:38.774022 16374 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cff7932ff3f1478d906e7efc868c2223: Generated new TSK 0
I20260810 23:07:38.774204 16374 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:38.781345 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 16277
2026-08-10T23:07:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1374 ms)
[----------] 18 tests from AuthTokenIssuingTest (24772 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:38 dist-test-slave-1xrh krb5kdc[16386](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:38 dist-test-slave-1xrh krb5kdc[16386](info): set up 2 sockets
Aug 10 23:07:38 dist-test-slave-1xrh krb5kdc[16386](info): commencing operation
krb5kdc: starting...
W20260810 23:07:39.872062 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:39 dist-test-slave-1xrh krb5kdc[16386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:39.989905 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:37921
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:37921,127.23.56.125:40863,127.23.56.124:44939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:40.072250 16400 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:07:40.072494 16400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:40.072525 16400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:40.072566 16400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:40.073686 16400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:40.073751 16400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:40.073778 16400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:40.073812 16400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:40.073837 16400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:40.075377 16400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:37921,127.23.56.125:40863,127.23.56.124:44939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:37921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:40.075650 16400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:40.075876 16400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:40.080643 16405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:40.080639 16409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:40.080703 16406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:40.080956 16400 server_base.cc:1034] running on GCE node
I20260810 23:07:40.081194 16400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:40.081507 16400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:40.082670 16400 hybrid_clock.cc:648] HybridClock initialized: now 1786403260082657 us; error 43 us; skew 500 ppm
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:40.085177 16400 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:40.085599 16400 webserver.cc:460] Webserver started at http://127.23.56.126:34265/ using document root <none> and password file <none>
I20260810 23:07:40.085811 16400 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:40.085862 16400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:40.086005 16400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:40.086983 16400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "355897b160d34c5c9e0c8b69978c6ccd"
format_stamp: "Formatted at 2026-08-10 23:07:40 on dist-test-slave-1xrh"
server_key: "2df9617e41b16b8a764cd184eb724ddc"
server_key_iv: "8646977ab033438da17b4e11e37cdcb8"
server_key_version: "encryptionkey@0"
I20260810 23:07:40.087373 16400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "355897b160d34c5c9e0c8b69978c6ccd"
format_stamp: "Formatted at 2026-08-10 23:07:40 on dist-test-slave-1xrh"
server_key: "2df9617e41b16b8a764cd184eb724ddc"
server_key_iv: "8646977ab033438da17b4e11e37cdcb8"
server_key_version: "encryptionkey@0"
I20260810 23:07:40.088649 16400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:07:40.089347 16416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:40.089564 16400 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:40.089632 16400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "355897b160d34c5c9e0c8b69978c6ccd"
format_stamp: "Formatted at 2026-08-10 23:07:40 on dist-test-slave-1xrh"
server_key: "2df9617e41b16b8a764cd184eb724ddc"
server_key_iv: "8646977ab033438da17b4e11e37cdcb8"
server_key_version: "encryptionkey@0"
I20260810 23:07:40.089718 16400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:40.105911 16400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:40.108162 16400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:40.108328 16400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:40.112159 16400 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:37921
I20260810 23:07:40.112211 16473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:37921 every 8 connection(s)
I20260810 23:07:40.112564 16400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:40.113483 16475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:40.115042 16475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:40.115347 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 16400
I20260810 23:07:40.115468 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:40.115691 23777 external_mini_cluster.cc:1428] Setting key 07d34b546b9b41a05c66fbaec15867f6
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.56.125@KRBTEST.COM; defaulting to no policy
I20260810 23:07:40.124898 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.115821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:37921 (local address 127.0.0.1:47184)
0810 23:07:40.116040 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.116076 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:40.116567 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:40.116860 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:40.116863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:40.117131 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:40.117353 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.117359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.118373 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.118374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.119065 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.119067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.119645 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.119646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.119705 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.120211 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:40.120225 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:40.121574 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:40.123426 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.123433 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.123435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.123649 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.123896 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.123897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.123897 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.123993 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.124335 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:40.124341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:40.124409 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 23:07:40.124500 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 23:07:40.124735 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"mutex_wait_us":98,"thread_start_us":113,"threads_started":1}
I20260810 23:07:40.124912 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.115827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47184 (local address 127.23.56.126:37921)
0810 23:07:40.116017 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.116028 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.116178 (+ 150us) server_negotiation.cc:407] Connection header received
0810 23:07:40.116725 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.116727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.116747 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.116854 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.117480 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.118276 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.119191 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.119512 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.119654 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.121672 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.121679 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.121679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.121685 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.123331 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.123715 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.123718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.123718 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.123805 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.124061 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.124062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.124063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.124263 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.124355 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.124534 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.124668 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":110,"threads_started":1}
I20260810 23:07:40.125672 16475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:40.125980 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.125870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:35752)
0810 23:07:40.125917 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.125960 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 23:07:40.126142 16419 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.125@KRBTEST.COM" created.
I20260810 23:07:40.126580 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.126483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:35764)
0810 23:07:40.126524 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.126547 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 23:07:40.126703 16475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.125:40863: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab.
Entry for principal kudu/127.23.56.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab.
I20260810 23:07:40.140053 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.125:40863
--webserver_interface=127.23.56.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab
--principal=kudu/127.23.56.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:37921,127.23.56.125:40863,127.23.56.124:44939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:07:40.175951 16475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } attempt: 1
I20260810 23:07:40.176432 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.176287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:35772)
0810 23:07:40.176349 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.176405 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 23:07:40.176795 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.176699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:35776)
0810 23:07:40.176744 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.176771 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 23:07:40.176882 16475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.125:40863: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
W20260810 23:07:40.221277 16482 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:07:40.221489 16482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:40.221508 16482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:40.221527 16482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:40.222922 16482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:40.222980 16482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:40.222994 16482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:40.223038 16482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:40.223070 16482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:40.224977 16482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:37921,127.23.56.125:40863,127.23.56.124:44939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab
--principal=kudu/127.23.56.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.125:40863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.56.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:40.225261 16482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:40.225510 16482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:40.230340 16489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:40.230341 16491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:40.230358 16488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:40.230583 16482 server_base.cc:1034] running on GCE node
I20260810 23:07:40.230787 16482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:40.231164 16482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:40.232350 16482 hybrid_clock.cc:648] HybridClock initialized: now 1786403260232329 us; error 30 us; skew 500 ppm
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:40.234874 16482 init.cc:377] Logged in from keytab as kudu/127.23.56.125@KRBTEST.COM (short username kudu)
I20260810 23:07:40.235363 16482 webserver.cc:460] Webserver started at http://127.23.56.125:46097/ using document root <none> and password file <none>
I20260810 23:07:40.235739 16482 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:40.235842 16482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:40.235950 16482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:40.236968 16482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/data/instance:
uuid: "8b40419ae17145f2865f0c77a5a10e04"
format_stamp: "Formatted at 2026-08-10 23:07:40 on dist-test-slave-1xrh"
server_key: "81c59e20b93f17fd551f8d32a7047f8c"
server_key_iv: "3b0c40fc2843172ec58574e8bced4725"
server_key_version: "encryptionkey@0"
I20260810 23:07:40.237283 16482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/wal/instance:
uuid: "8b40419ae17145f2865f0c77a5a10e04"
format_stamp: "Formatted at 2026-08-10 23:07:40 on dist-test-slave-1xrh"
server_key: "81c59e20b93f17fd551f8d32a7047f8c"
server_key_iv: "3b0c40fc2843172ec58574e8bced4725"
server_key_version: "encryptionkey@0"
I20260810 23:07:40.238523 16482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:40.239290 16500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:40.239459 16482 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 23:07:40.239547 16482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/wal
uuid: "8b40419ae17145f2865f0c77a5a10e04"
format_stamp: "Formatted at 2026-08-10 23:07:40 on dist-test-slave-1xrh"
server_key: "81c59e20b93f17fd551f8d32a7047f8c"
server_key_iv: "3b0c40fc2843172ec58574e8bced4725"
server_key_version: "encryptionkey@0"
I20260810 23:07:40.239602 16482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:40.245015 16475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } attempt: 2
I20260810 23:07:40.245409 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.245276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:35780)
0810 23:07:40.245338 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.245376 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260810 23:07:40.245784 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.245696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:35786)
0810 23:07:40.245741 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.245766 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260810 23:07:40.245889 16475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.125:40863: Network error: Client connection negotiation failed: client connection to 127.23.56.125:40863: connect: Connection refused (error 111)
I20260810 23:07:40.269972 16482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:40.272497 16482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:40.272663 16482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:40.276660 16482 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.125:40863
I20260810 23:07:40.276700 16559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.125:40863 every 8 connection(s)
I20260810 23:07:40.277129 16482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
I20260810 23:07:40.278136 16560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:40.279757 16560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:40.284667 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 16482
I20260810 23:07:40.284891 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-1/wal/instance
I20260810 23:07:40.285082 23777 external_mini_cluster.cc:1428] Setting key abefb40a93153dd77f35a7188d2e55a6
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:40.288507 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.280460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.23.56.126:37921)
0810 23:07:40.280550 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.280553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.280732 (+ 179us) server_negotiation.cc:407] Connection header received
0810 23:07:40.281159 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.281160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.281173 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.281224 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.281765 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.282244 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.283245 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.283448 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.283511 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.285710 (+ 2199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.285712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.285713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.285719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.287065 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.287564 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.287565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.287565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.287618 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.287897 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.287898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.287899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.288008 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.288060 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.288374 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.288445 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 23:07:40.288687 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.280407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:37921 (local address 127.0.0.1:47194)
0810 23:07:40.280615 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.280651 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:40.281078 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:40.281274 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:40.281277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:40.281515 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:40.281696 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.281699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.282319 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.282320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.283084 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.283086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.283538 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.283539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.283651 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.284081 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:40.284088 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:40.285616 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:40.287164 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.287172 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.287174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.287488 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.287689 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.287690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.287691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.287830 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.288081 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:40.288088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:40.288246 (+ 158us) client_negotiation.cc:769] Sending connection context
0810 23:07:40.288348 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 23:07:40.288464 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":126,"threads_started":1}
I20260810 23:07:40.289245 16560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.56.124@KRBTEST.COM; defaulting to no policy
I20260810 23:07:40.297048 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.289500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:35792)
0810 23:07:40.289549 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.289559 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:40.289598 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:40.289824 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:40.289825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:40.289962 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:40.290005 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.290006 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.290900 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.290900 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.291806 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.291809 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.292386 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.292387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.292479 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.292815 (+ 336us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:40.292822 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:40.294110 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:40.295919 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.295919 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.295920 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.296081 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.296307 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.296307 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.296308 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.296361 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.296659 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:40.296660 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:40.296698 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:07:40.296769 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 23:07:40.296866 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260810 23:07:40.297060 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.289580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35792 (local address 127.23.56.125:40863)
0810 23:07:40.289704 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.289708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.289721 (+ 13us) server_negotiation.cc:407] Connection header received
0810 23:07:40.289736 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.289737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.289747 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.289823 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.290089 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.290790 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.291892 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.292270 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.292376 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.294195 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.294204 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.294205 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.294211 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.295807 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.296182 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.296185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.296186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.296242 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.296448 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.296449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.296449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.296590 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.296681 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.296800 (+ 119us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.296942 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":103,"thread_start_us":82,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.124@KRBTEST.COM" created.
I20260810 23:07:40.297741 16560 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:40.298080 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.297981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36540)
0810 23:07:40.298016 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.298059 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260810 23:07:40.298244 16501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
I20260810 23:07:40.298638 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.298552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36544)
0810 23:07:40.298597 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.298621 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 23:07:40.298768 16560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:44939: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab.
Entry for principal kudu/127.23.56.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab.
I20260810 23:07:40.313802 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.124:44939
--webserver_interface=127.23.56.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab
--principal=kudu/127.23.56.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:42235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:37921,127.23.56.125:40863,127.23.56.124:44939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:07:40.337064 16475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } attempt: 3
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
I20260810 23:07:40.344736 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.337466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:35800)
0810 23:07:40.337531 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.337543 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:40.337609 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:40.337764 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:40.337765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:40.337940 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:40.338015 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.338017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.338716 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.338718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.339299 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.339302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.339719 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.339720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.339804 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.340259 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:40.340268 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:40.341782 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:40.343802 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.343803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.343803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.343970 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.344187 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.344187 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.344187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.344238 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.344508 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:40.344509 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:40.344547 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 23:07:40.344598 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 23:07:40.344669 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 23:07:40.344820 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.337539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35800 (local address 127.23.56.125:40863)
0810 23:07:40.337609 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.337612 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.337623 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:07:40.337673 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.337674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.337685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.337756 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.338113 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.338630 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.339411 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.339631 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.339720 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.341879 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.341881 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.341881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.341888 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.343706 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.344047 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.344048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.344048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.344117 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.344304 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.344304 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.344305 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.344444 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.344516 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.344644 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.344729 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:07:40.345189 16475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:40.345542 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.345436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36560)
0810 23:07:40.345486 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.345517 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:07:40.345896 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.345812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36568)
0810 23:07:40.345852 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.345875 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260810 23:07:40.345993 16475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:44939: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
I20260810 23:07:40.347975 16560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } attempt: 1
I20260810 23:07:40.348286 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.348181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36570)
0810 23:07:40.348230 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.348256 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 23:07:40.348577 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.348492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36574)
0810 23:07:40.348534 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.348556 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 23:07:40.348671 16560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:44939: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
I20260810 23:07:40.377146 16475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } attempt: 1
I20260810 23:07:40.377592 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.377463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36582)
0810 23:07:40.377526 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.377564 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 23:07:40.377933 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.377826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36594)
0810 23:07:40.377870 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.377896 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 23:07:40.378019 16475 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:44939: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
W20260810 23:07:40.399305 16567 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:07:40.399533 16567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:40.399574 16567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:40.399614 16567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:40.400686 16567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:40.400748 16567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:40.400771 16567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:40.400796 16567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:40.400815 16567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:40.402222 16567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:42235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:37921,127.23.56.125:40863,127.23.56.124:44939
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab
--principal=kudu/127.23.56.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.124:44939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.56.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:40.402487 16567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:40.402705 16567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:40.407536 16576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:40.407572 16573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:40.407783 16567 server_base.cc:1034] running on GCE node
W20260810 23:07:40.407572 16574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:40.408094 16567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:40.408465 16567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:40.409639 16567 hybrid_clock.cc:648] HybridClock initialized: now 1786403260409621 us; error 37 us; skew 500 ppm
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:40.412153 16567 init.cc:377] Logged in from keytab as kudu/127.23.56.124@KRBTEST.COM (short username kudu)
I20260810 23:07:40.412566 16567 webserver.cc:460] Webserver started at http://127.23.56.124:45489/ using document root <none> and password file <none>
I20260810 23:07:40.412777 16567 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:40.412844 16567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:40.412997 16567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:40.414068 16567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/data/instance:
uuid: "787a66d79c28464eb20ebd2d0ac80b46"
format_stamp: "Formatted at 2026-08-10 23:07:40 on dist-test-slave-1xrh"
server_key: "e4a3ccbd9aae548eca1c51297c8392fd"
server_key_iv: "09ccf68b26f58cdf4ebfe1e05d976de2"
server_key_version: "encryptionkey@0"
I20260810 23:07:40.414530 16567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/wal/instance:
uuid: "787a66d79c28464eb20ebd2d0ac80b46"
format_stamp: "Formatted at 2026-08-10 23:07:40 on dist-test-slave-1xrh"
server_key: "e4a3ccbd9aae548eca1c51297c8392fd"
server_key_iv: "09ccf68b26f58cdf4ebfe1e05d976de2"
server_key_version: "encryptionkey@0"
I20260810 23:07:40.415984 16567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260810 23:07:40.416663 16583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:40.416819 16560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } attempt: 2
I20260810 23:07:40.416885 16567 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:40.417006 16567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/wal
uuid: "787a66d79c28464eb20ebd2d0ac80b46"
format_stamp: "Formatted at 2026-08-10 23:07:40 on dist-test-slave-1xrh"
server_key: "e4a3ccbd9aae548eca1c51297c8392fd"
server_key_iv: "09ccf68b26f58cdf4ebfe1e05d976de2"
server_key_version: "encryptionkey@0"
I20260810 23:07:40.417111 16567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:40.417219 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.417074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36606)
0810 23:07:40.417141 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.417183 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 23:07:40.417563 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.417465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36620)
0810 23:07:40.417512 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.417540 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 23:07:40.417686 16560 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:44939: Network error: Client connection negotiation failed: client connection to 127.23.56.124:44939: connect: Connection refused (error 111)
I20260810 23:07:40.433827 16567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:40.435736 16567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:40.435894 16567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:40.439990 16567 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.124:44939
I20260810 23:07:40.440038 16643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.124:44939 every 8 connection(s)
I20260810 23:07:40.440394 16567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
I20260810 23:07:40.441345 16644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:40.442916 16644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:40.448911 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 16567
I20260810 23:07:40.449045 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786402992661514-23777-0/minicluster-data/master-2/wal/instance
I20260810 23:07:40.449231 23777 external_mini_cluster.cc:1428] Setting key ce89e697b0847ea4e0367b0356a9b8d7
I20260810 23:07:40.451958 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.443665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.23.56.126:37921)
0810 23:07:40.443736 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.443754 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.443955 (+ 201us) server_negotiation.cc:407] Connection header received
0810 23:07:40.444485 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.444486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.444499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.444551 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.445159 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.445632 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.446568 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.446775 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.446852 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.449009 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.449011 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.449011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.449018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.450591 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.450994 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.450995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.450995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.451051 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.451330 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.451331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.451332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.451450 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.451526 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.451782 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.451860 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:07:40.452117 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.443623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:37921 (local address 127.0.0.1:47200)
0810 23:07:40.443840 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.443874 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:40.444352 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:40.444621 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:40.444624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:40.444864 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:40.445054 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.445058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.445724 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.445725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.446477 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.446479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.446870 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.446871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.447015 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.447576 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:40.447585 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:40.448910 (+ 1325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:40.450693 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.450700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.450703 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.450916 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.451151 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.451151 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.451152 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.451261 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.451518 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:40.451525 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:40.451656 (+ 131us) client_negotiation.cc:769] Sending connection context
0810 23:07:40.451759 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 23:07:40.451900 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
I20260810 23:07:40.452740 16644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:40.453131 16475 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } attempt: 2
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
I20260810 23:07:40.458436 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.449676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.23.56.124:44939)
0810 23:07:40.449842 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.449847 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.449918 (+ 71us) server_negotiation.cc:407] Connection header received
0810 23:07:40.449973 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.449976 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.449989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.450111 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.450696 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.451415 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.452184 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.452533 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.452617 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.455164 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.455173 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.455173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.455180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.457095 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.457456 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.457459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.457459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.457519 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.457748 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.457749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.457750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.457937 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.458078 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.458219 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.458304 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260810 23:07:40.461072 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.453009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:40863 (local address 127.0.0.1:35810)
0810 23:07:40.453062 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.453072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:40.453106 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:40.453314 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:40.453315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:40.453486 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:40.453547 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.453549 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.454354 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.454355 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.455069 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.455071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.455628 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.455628 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.455690 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.456083 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:40.456088 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:40.457793 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:40.459861 (+ 2068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.459862 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.459862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.460172 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.460408 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.460409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.460409 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.460472 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.460761 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:40.460762 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:40.460809 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 23:07:40.460893 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 23:07:40.460992 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260810 23:07:40.461120 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.453098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.23.56.125:40863)
0810 23:07:40.453142 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.453144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.453152 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:40.453177 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.453179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.453193 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.453264 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.453650 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.454267 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.455189 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.455515 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.455558 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.457890 (+ 2332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.457891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.457892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.457899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.459779 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.460265 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.460267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.460267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.460330 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.460547 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.460548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.460549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.460677 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.460728 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.460944 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.461019 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:07:40.461845 16644 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:40.461928 16478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.453357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36644)
0810 23:07:40.453401 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.453410 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:40.453446 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:40.453659 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:40.453660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:40.453833 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:40.453877 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.453879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.454753 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.454754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.455531 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.455534 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.456041 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.456042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.456098 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.456415 (+ 317us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:40.456421 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:40.457839 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:40.460768 (+ 2929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.460768 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.460769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.460973 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.461205 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.461206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.461206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.461268 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.461605 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:40.461606 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:40.461652 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 23:07:40.461722 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 23:07:40.461818 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 23:07:40.462024 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.453397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.23.56.124:44939)
0810 23:07:40.453526 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.453528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.453536 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:40.453553 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.453554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.453563 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.453634 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.453970 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.454645 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.455622 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.455920 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.456037 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.457930 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.457931 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.457931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.457937 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.460474 (+ 2537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.461056 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.461057 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.461057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.461111 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.461345 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.461345 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.461346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.461441 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.461526 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.461829 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.461899 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260810 23:07:40.463917 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd: Bootstrap starting.
I20260810 23:07:40.464598 16475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:40.464928 16475 log.cc:824] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd: Log is configured to *not* fsync() on all Append() calls
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:40.466017 16475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd: No bootstrap required, opened a new log
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
I20260810 23:07:40.468199 16475 raft_consensus.cc:348] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } }
I20260810 23:07:40.468292 16475 raft_consensus.cc:729] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 355897b160d34c5c9e0c8b69978c6ccd, State: Initialized, Role: FOLLOWER
I20260810 23:07:40.468343 16475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } }
I20260810 23:07:40.468578 16652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:40.468650 16475 sys_catalog.cc:564] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:40.468654 16652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:40.468969 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.460251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47212 (local address 127.23.56.126:37921)
0810 23:07:40.460287 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.460288 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.460314 (+ 26us) server_negotiation.cc:407] Connection header received
0810 23:07:40.460334 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.460334 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.460344 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.460406 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.460991 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.461680 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.462694 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.462961 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.463010 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.465588 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.465590 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.465590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.465597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.467095 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.467512 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.467514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.467514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.467567 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.467779 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.467781 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.467781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.467928 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.468085 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.468762 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.468841 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:07:40.469689 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.462062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36648)
0810 23:07:40.462453 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.462463 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:40.462513 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:40.462630 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:40.462631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:40.462774 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:40.462802 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.462804 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.463720 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.463722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.464574 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.464577 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.464945 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.464946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.464995 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.465384 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:40.465390 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:40.466737 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:40.468556 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.468557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.468558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.468769 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.468952 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.468952 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.468953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.468998 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.469410 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:40.469411 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:40.469452 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:07:40.469526 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 23:07:40.469606 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376}
I20260810 23:07:40.469918 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.462127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36648 (local address 127.23.56.124:44939)
0810 23:07:40.462156 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.462158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.462518 (+ 360us) server_negotiation.cc:407] Connection header received
0810 23:07:40.462563 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.462563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.462573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.462626 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.462879 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.463628 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.464641 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.464855 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.464943 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.466833 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.466834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.466834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.466841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.468456 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.468844 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.468845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.468845 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.468890 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.469062 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.469062 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.469063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.469207 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.469361 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.469738 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.469821 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 23:07:40.470988 16663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:07:40.471098 16663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:07:40.471490 16644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46: Bootstrap starting.
I20260810 23:07:40.472127 16644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:40.472414 16644 log.cc:824] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:40.473186 16644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46: No bootstrap required, opened a new log
I20260810 23:07:40.471704 16400 master_runner.cc:186] Error getting master registration for 127.23.56.125:40863: OK, instance_id { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" instance_seqno: 1786403260275543 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:07:40.474669 16400 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
I20260810 23:07:40.475215 16644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } }
I20260810 23:07:40.475345 16644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 787a66d79c28464eb20ebd2d0ac80b46, State: Initialized, Role: FOLLOWER
I20260810 23:07:40.475426 16644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } }
I20260810 23:07:40.475708 16644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:40.475697 16666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:40.477526 16666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:07:40.478025 16677 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:07:40.478124 16677 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:07:40.478820 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.469809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35824 (local address 127.23.56.125:40863)
0810 23:07:40.469848 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.469850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.469858 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:40.469946 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.469948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.469958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.469990 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.470692 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.471378 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.472319 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.472673 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.472713 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.475291 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.475293 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.475293 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.475300 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.477079 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.477769 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.477770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.477770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.477833 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.478047 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.478048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.478048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.478173 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.478301 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.478628 (+ 327us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.478714 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:07:40.480376 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:40.479087 16567 master_runner.cc:186] Error getting master registration for 127.23.56.125:40863: OK, instance_id { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" instance_seqno: 1786403260275543 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:07:40.481256 16567 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:07:40.481725 16445 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "355897b160d34c5c9e0c8b69978c6ccd"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47212
I20260810 23:07:40.481814 16445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:07:40.481843 16445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:40.482568 16445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } }
I20260810 23:07:40.482805 16445 leader_election.cc:288] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [CANDIDATE]: Term 1 election: Requested vote from peers 8b40419ae17145f2865f0c77a5a10e04 (127.23.56.125:40863), 787a66d79c28464eb20ebd2d0ac80b46 (127.23.56.124:44939)
I20260810 23:07:40.483249 16613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "355897b160d34c5c9e0c8b69978c6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "787a66d79c28464eb20ebd2d0ac80b46"
I20260810 23:07:40.483367 16613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:40.484321 16613 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 355897b160d34c5c9e0c8b69978c6ccd in term 1.
I20260810 23:07:40.484517 16417 leader_election.cc:302] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 355897b160d34c5c9e0c8b69978c6ccd, 787a66d79c28464eb20ebd2d0ac80b46; no voters:
I20260810 23:07:40.484648 16652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:40.484822 16652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [term 1 LEADER]: Becoming Leader. State: Replica: 355897b160d34c5c9e0c8b69978c6ccd, State: Running, Role: LEADER
I20260810 23:07:40.484923 16652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } }
I20260810 23:07:40.486352 16685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355897b160d34c5c9e0c8b69978c6ccd. Latest consensus state: current_term: 1 leader_uuid: "355897b160d34c5c9e0c8b69978c6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:40.486521 16685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:40.486769 16687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:40.486987 16687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:40.488498 16613 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [term 1 FOLLOWER]: Refusing update from remote peer 355897b160d34c5c9e0c8b69978c6ccd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:40.488853 16685 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [LEADER]: Connected to new peer: Peer: permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:40.490031 16666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355897b160d34c5c9e0c8b69978c6ccd. Latest consensus state: current_term: 1 leader_uuid: "355897b160d34c5c9e0c8b69978c6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:40.490566 16666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:40.492089 16685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "355897b160d34c5c9e0c8b69978c6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:40.492167 16685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:40.492568 16666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "355897b160d34c5c9e0c8b69978c6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:40.492655 16666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:40.492959 16688 mvcc.cc:205] Tried to move back new op lower bound from 7317107754958282752 to 7317107754947117056. Current Snapshot: MvccSnapshot[applied={T|T < 7317107754958282752}]
I20260810 23:07:40.493448 16477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.484022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.23.56.126:37921)
0810 23:07:40.484443 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.484445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.484454 (+ 9us) server_negotiation.cc:407] Connection header received
0810 23:07:40.484474 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.484475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.484484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.484539 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.485355 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.486068 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.487607 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.487896 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.487935 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.490109 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.490111 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.490122 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.490129 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.491833 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.492391 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.492392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.492393 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.492460 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.492760 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.492762 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.492762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.492887 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.492973 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.493276 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.493347 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378}
I20260810 23:07:40.494266 16689 mvcc.cc:205] Tried to move back new op lower bound from 7317107754958282752 to 7317107754947117056. Current Snapshot: MvccSnapshot[applied={T|T < 7317107754958282752}]
I20260810 23:07:40.494692 16687 catalog_manager.cc:1347] Generated new cluster ID: 87a45cf9a7c142d79485664edb653b73
I20260810 23:07:40.494740 16687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:40.505806 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.495349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36664 (local address 127.23.56.124:44939)
0810 23:07:40.495413 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.495414 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.495563 (+ 149us) server_negotiation.cc:407] Connection header received
0810 23:07:40.495788 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.495789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.495800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.495832 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.496507 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.497298 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.498201 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.498504 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.498547 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.500084 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.500086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.500086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.500093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.502048 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.503539 (+ 1491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.503540 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.503540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.503601 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.504858 (+ 1257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.504859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.504859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.505005 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.505122 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.505625 (+ 503us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.505704 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 23:07:40.507241 16564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.495475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35838 (local address 127.23.56.125:40863)
0810 23:07:40.495522 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.495525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.495890 (+ 365us) server_negotiation.cc:407] Connection header received
0810 23:07:40.496026 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.496027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.496052 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.496083 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.497271 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.497966 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.499771 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.500614 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.500674 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.503023 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.503025 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.503026 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.503032 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.504765 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.506251 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.506252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.506253 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.506332 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.506577 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.506578 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.506578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.506711 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.506792 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.507011 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.507101 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260810 23:07:40.508818 16560 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } attempt: 3
I20260810 23:07:40.510526 16687 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:40.511245 16687 catalog_manager.cc:1504] Loading token signing keys...
Aug 10 23:07:40 dist-test-slave-1xrh krb5kdc[16386](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403260, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
I20260810 23:07:40.517041 16562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.509039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:44939 (local address 127.0.0.1:36674)
0810 23:07:40.509083 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:40.509092 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:40.509127 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:40.509310 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:40.509310 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:40.509472 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:40.509521 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.509523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.510328 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.510329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.511155 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:40.511158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.511617 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.511618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:40.511674 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.512117 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:40.512122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:40.513606 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:40.515786 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.515787 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.515787 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.516052 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.516282 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:40.516283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:40.516283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:40.516335 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:40.516741 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:40.516741 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:40.516786 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 23:07:40.516835 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 23:07:40.516941 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 23:07:40.517360 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:40.509088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36674 (local address 127.23.56.124:44939)
0810 23:07:40.509191 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.509192 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.509199 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:07:40.509217 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.509218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.509226 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.509271 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.509605 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.510238 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.511242 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:40.511542 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:40.511617 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:40.513688 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:40.513690 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:40.513690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:40.513697 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:40.515700 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.516141 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.516142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.516142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.516201 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:40.516409 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:40.516410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:40.516410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:40.516535 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:40.517154 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:40.517201 (+ 47us) server_negotiation.cc:299] Negotiation successful
0810 23:07:40.517283 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260810 23:07:40.517946 16687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd: Generated new TSK 0
I20260810 23:07:40.518298 16687 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:40.518965 16560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04: Bootstrap starting.
I20260810 23:07:40.519696 16560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:40.520023 16560 log.cc:824] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:40.521052 16560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04: No bootstrap required, opened a new log
I20260810 23:07:40.523080 16560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } }
I20260810 23:07:40.523361 16560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b40419ae17145f2865f0c77a5a10e04, State: Initialized, Role: FOLLOWER
I20260810 23:07:40.523488 16560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } }
I20260810 23:07:40.523869 16560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:40.523975 16528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:40.524870 16528 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [term 1 FOLLOWER]: Refusing update from remote peer 355897b160d34c5c9e0c8b69978c6ccd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:40.525060 16529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "355897b160d34c5c9e0c8b69978c6ccd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b40419ae17145f2865f0c77a5a10e04"
I20260810 23:07:40.525163 16685 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:40.525811 16529 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 355897b160d34c5c9e0c8b69978c6ccd in term 1.
I20260810 23:07:40.528126 16710 mvcc.cc:205] Tried to move back new op lower bound from 7317107755077705728 to 7317107754947117056. Current Snapshot: MvccSnapshot[applied={T|T < 7317107754958282752}]
I20260810 23:07:40.528280 16712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "355897b160d34c5c9e0c8b69978c6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:40.528398 16712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:40.528563 16712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 355897b160d34c5c9e0c8b69978c6ccd. Latest consensus state: current_term: 1 leader_uuid: "355897b160d34c5c9e0c8b69978c6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:40.528637 16712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:40.528681 16712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "355897b160d34c5c9e0c8b69978c6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:40.528734 16712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04 [sys.catalog]: This master's current role is: FOLLOWER
W20260810 23:07:40.530889 16713 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:07:40.530977 16713 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 23:07:40.556257 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.554565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.23.56.125:40863)
0810 23:07:40.554610 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.554613 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.555600 (+ 987us) server_negotiation.cc:407] Connection header received
0810 23:07:40.555706 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.555708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.555719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.555752 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.556187 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35852: BlockingRecv error: recv got EOF from 127.0.0.1:35852 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 23:07:40.557206 16477 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.554979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47240 (local address 127.23.56.126:37921)
0810 23:07:40.555019 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.555021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.556637 (+ 1616us) server_negotiation.cc:407] Connection header received
0810 23:07:40.556737 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.556738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.556752 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.556787 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.557141 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47240: BlockingRecv error: recv got EOF from 127.0.0.1:47240 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 23:07:40.557989 16650 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.555333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36684 (local address 127.23.56.124:44939)
0810 23:07:40.555376 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.555379 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.557509 (+ 2130us) server_negotiation.cc:407] Connection header received
0810 23:07:40.557604 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.557605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.557615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.557645 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.557930 (+ 285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36684: BlockingRecv error: recv got EOF from 127.0.0.1:36684 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260810 23:07:40.560009 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.559294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.23.56.125:40863)
0810 23:07:40.559337 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.559338 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.559489 (+ 151us) server_negotiation.cc:407] Connection header received
0810 23:07:40.559594 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.559595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.559613 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.559642 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.559956 (+ 314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35868: BlockingRecv error: recv got EOF from 127.0.0.1:35868 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260810 23:07:40.562956 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.562226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.23.56.125:40863)
0810 23:07:40.562260 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.562263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.562504 (+ 241us) server_negotiation.cc:407] Connection header received
0810 23:07:40.562595 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.562596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.562605 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.562632 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.562914 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35880: BlockingRecv error: recv got EOF from 127.0.0.1:35880 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 23:07:40.567016 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.566061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35892 (local address 127.23.56.125:40863)
0810 23:07:40.566092 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.566093 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.566308 (+ 215us) server_negotiation.cc:407] Connection header received
0810 23:07:40.566533 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.566534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.566548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.566591 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.566975 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35892: BlockingRecv error: recv got EOF from 127.0.0.1:35892 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260810 23:07:40.572831 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.572077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35904 (local address 127.23.56.125:40863)
0810 23:07:40.572109 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.572111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.572339 (+ 228us) server_negotiation.cc:407] Connection header received
0810 23:07:40.572430 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.572431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.572440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.572468 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.572791 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35904: BlockingRecv error: recv got EOF from 127.0.0.1:35904 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 23:07:40.582819 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.581953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35914 (local address 127.23.56.125:40863)
0810 23:07:40.581989 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.581991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.582264 (+ 273us) server_negotiation.cc:407] Connection header received
0810 23:07:40.582369 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.582370 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.582381 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.582411 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.582748 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35914: BlockingRecv error: recv got EOF from 127.0.0.1:35914 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 23:07:40.601080 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.600134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.23.56.125:40863)
0810 23:07:40.600173 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.600175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.600481 (+ 306us) server_negotiation.cc:407] Connection header received
0810 23:07:40.600589 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.600590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.600602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.600634 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.601028 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35924: BlockingRecv error: recv got EOF from 127.0.0.1:35924 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 23:07:40.635533 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.634527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.23.56.125:40863)
0810 23:07:40.634573 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.634575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.634899 (+ 324us) server_negotiation.cc:407] Connection header received
0810 23:07:40.635008 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.635009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.635022 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.635053 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.635472 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35926: BlockingRecv error: recv got EOF from 127.0.0.1:35926 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260810 23:07:40.701617 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.700953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.23.56.125:40863)
0810 23:07:40.701007 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.701009 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.701120 (+ 111us) server_negotiation.cc:407] Connection header received
0810 23:07:40.701182 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.701183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.701195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.701257 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.701552 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35940: BlockingRecv error: recv got EOF from 127.0.0.1:35940 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260810 23:07:40.831497 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:40.830719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35954 (local address 127.23.56.125:40863)
0810 23:07:40.830815 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:40.830817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:40.830879 (+ 62us) server_negotiation.cc:407] Connection header received
0810 23:07:40.830923 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:40.830924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:40.830937 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:40.830988 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:40.831400 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35954: BlockingRecv error: recv got EOF from 127.0.0.1:35954 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 23:07:41.087165 16685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "355897b160d34c5c9e0c8b69978c6ccd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "355897b160d34c5c9e0c8b69978c6ccd" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 37921 } } peers { permanent_uuid: "8b40419ae17145f2865f0c77a5a10e04" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 40863 } } peers { permanent_uuid: "787a66d79c28464eb20ebd2d0ac80b46" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 44939 } } }
I20260810 23:07:41.087347 16685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 355897b160d34c5c9e0c8b69978c6ccd [sys.catalog]: This master's current role is: LEADER
W20260810 23:07:41.089192 16564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:41.088576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35956 (local address 127.23.56.125:40863)
0810 23:07:41.088650 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:41.088652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:41.088711 (+ 59us) server_negotiation.cc:407] Connection header received
0810 23:07:41.088765 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:41.088766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:41.088777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:41.088817 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:41.089139 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35956: BlockingRecv error: recv got EOF from 127.0.0.1:35956 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 23:07:41.478607 16677 catalog_manager.cc:1259] Loaded cluster ID: 87a45cf9a7c142d79485664edb653b73
I20260810 23:07:41.478682 16677 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46: loading cluster ID for follower catalog manager: success
I20260810 23:07:41.479720 16677 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46: acquiring CA information for follower catalog manager: success
I20260810 23:07:41.479993 16677 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 787a66d79c28464eb20ebd2d0ac80b46: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:07:41.531448 16713 catalog_manager.cc:1259] Loaded cluster ID: 87a45cf9a7c142d79485664edb653b73
I20260810 23:07:41.531523 16713 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04: loading cluster ID for follower catalog manager: success
I20260810 23:07:41.532461 16713 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04: acquiring CA information for follower catalog manager: success
I20260810 23:07:41.532920 16713 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b40419ae17145f2865f0c77a5a10e04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:07:41.604861 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:41.602375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.23.56.125:40863)
0810 23:07:41.602525 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:41.602527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:41.602542 (+ 15us) server_negotiation.cc:407] Connection header received
0810 23:07:41.602619 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:41.602619 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:41.602673 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:41.602758 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:41.602906 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.603439 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.604182 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.604376 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.604428 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:41.604574 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:41.604683 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:41.604740 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:41.604775 (+ 35us) server_negotiation.cc:299] Negotiation successful
0810 23:07:41.604793 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260810 23:07:41.609016 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:41.606678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.23.56.124:44939)
0810 23:07:41.606776 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:41.606778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:41.606803 (+ 25us) server_negotiation.cc:407] Connection header received
0810 23:07:41.606839 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:41.606839 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:41.606873 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:41.606937 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:41.607223 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.607706 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.608351 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.608540 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.608588 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:41.608712 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:41.608796 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:41.608847 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:41.608883 (+ 36us) server_negotiation.cc:299] Negotiation successful
0810 23:07:41.608900 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":50,"threads_started":1}
I20260810 23:07:41.613467 16808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:41.610667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.23.56.124:44939)
0810 23:07:41.610719 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:41.610721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:41.610901 (+ 180us) server_negotiation.cc:407] Connection header received
0810 23:07:41.611002 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:41.611003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:41.611014 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:41.611045 (+ 31us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:41.611249 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.611960 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.612743 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.613026 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.613081 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:41.613208 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:41.613237 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:41.613287 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:41.613371 (+ 84us) server_negotiation.cc:299] Negotiation successful
0810 23:07:41.613391 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 23:07:41.613875 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:41.610609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47252 (local address 127.23.56.126:37921)
0810 23:07:41.610719 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:41.610721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:41.610983 (+ 262us) server_negotiation.cc:407] Connection header received
0810 23:07:41.611050 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:41.611051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:41.611104 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:41.611209 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:41.611348 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.611844 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.612875 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.613223 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.613291 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:41.613501 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:41.613607 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:41.613646 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:41.613763 (+ 117us) server_negotiation.cc:299] Negotiation successful
0810 23:07:41.613792 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
I20260810 23:07:41.614033 16802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:41.610607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.23.56.125:40863)
0810 23:07:41.610645 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:41.610647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:41.610925 (+ 278us) server_negotiation.cc:407] Connection header received
0810 23:07:41.611037 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:41.611037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:41.611048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:41.611074 (+ 26us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:41.611299 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.612007 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.613022 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:41.613316 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:41.613384 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:41.613761 (+ 377us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:41.613795 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:41.613843 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:41.613924 (+ 81us) server_negotiation.cc:299] Negotiation successful
0810 23:07:41.613954 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:07:41.615716 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 16400
I20260810 23:07:41.628563 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 16482
I20260810 23:07:41.634886 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 16567
2026-08-10T23:07:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2851 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 23:07:41 dist-test-slave-1xrh krb5kdc[16821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 23:07:41 dist-test-slave-1xrh krb5kdc[16821](info): set up 2 sockets
Aug 10 23:07:41 dist-test-slave-1xrh krb5kdc[16821](info): commencing operation
krb5kdc: starting...
W20260810 23:07:42.722365 23777 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/test-user.keytab.
Aug 10 23:07:42 dist-test-slave-1xrh krb5kdc[16821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T23:07:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T23:07:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.56.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
Entry for principal kudu/127.23.56.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab.
I20260810 23:07:42.842156 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:39999
--webserver_interface=127.23.56.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:39999,127.23.56.125:43417,127.23.56.124:37525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 23:07:42.924015 16834 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:07:42.924219 16834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:42.924237 16834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:42.924256 16834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:42.925294 16834 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:42.925326 16834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:42.925340 16834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:42.925352 16834 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:42.925364 16834 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:42.926783 16834 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:39999,127.23.56.125:43417,127.23.56.124:37525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.126.keytab
--principal=kudu/127.23.56.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.126:39999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.56.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:42.926982 16834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:42.927217 16834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:42.931993 16841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:42.932020 16843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:42.932070 16845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:42.932211 16834 server_base.cc:1034] running on GCE node
I20260810 23:07:42.932432 16834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:42.932710 16834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:42.933892 16834 hybrid_clock.cc:648] HybridClock initialized: now 1786403262933864 us; error 57 us; skew 500 ppm
Aug 10 23:07:42 dist-test-slave-1xrh krb5kdc[16821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403262, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:42.936488 16834 init.cc:377] Logged in from keytab as kudu/127.23.56.126@KRBTEST.COM (short username kudu)
I20260810 23:07:42.936996 16834 webserver.cc:460] Webserver started at http://127.23.56.126:43643/ using document root <none> and password file <none>
I20260810 23:07:42.937242 16834 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:42.937311 16834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:42.937546 16834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:42.938596 16834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/data/instance:
uuid: "5122cb5c44274f288700b415081ecf7f"
format_stamp: "Formatted at 2026-08-10 23:07:42 on dist-test-slave-1xrh"
server_key: "10a0ee515e1fcd724de09afd02f93dfb"
server_key_iv: "876e719d66ebec412343c1ab4d274361"
server_key_version: "encryptionkey@0"
I20260810 23:07:42.938961 16834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/wal/instance:
uuid: "5122cb5c44274f288700b415081ecf7f"
format_stamp: "Formatted at 2026-08-10 23:07:42 on dist-test-slave-1xrh"
server_key: "10a0ee515e1fcd724de09afd02f93dfb"
server_key_iv: "876e719d66ebec412343c1ab4d274361"
server_key_version: "encryptionkey@0"
I20260810 23:07:42.940366 16834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 23:07:42.941074 16853 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:42.941239 16834 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:42.941339 16834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/wal
uuid: "5122cb5c44274f288700b415081ecf7f"
format_stamp: "Formatted at 2026-08-10 23:07:42 on dist-test-slave-1xrh"
server_key: "10a0ee515e1fcd724de09afd02f93dfb"
server_key_iv: "876e719d66ebec412343c1ab4d274361"
server_key_version: "encryptionkey@0"
I20260810 23:07:42.941423 16834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:42.964049 16834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:42.966456 16834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:42.966624 16834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:42.970940 16834 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.126:39999
I20260810 23:07:42.970975 16911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.126:39999 every 8 connection(s)
I20260810 23:07:42.971364 16834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/data/info.pb
I20260810 23:07:42.972327 16912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:42.973811 16912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:42.976946 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 16834
I20260810 23:07:42.977034 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-0/wal/instance
I20260810 23:07:42.977247 23777 external_mini_cluster.cc:1428] Setting key 3a8ac47b7435e75867cab0d728d317d1
Aug 10 23:07:42 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403262, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:42.983868 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:42.974430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:39999 (local address 127.0.0.1:57936)
0810 23:07:42.974606 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:42.974648 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:42.975088 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:42.975417 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:42.975420 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:42.975649 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:42.975798 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:42.975802 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:42.976779 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:42.976781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:42.977605 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:42.977607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:42.978347 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:42.978349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:42.978426 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:42.979113 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:42.979165 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:42.980531 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:42.982295 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:42.982302 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:42.982304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:42.982500 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:42.982749 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:42.982750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:42.982750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:42.982838 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:42.983243 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:42.983248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:42.983314 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 23:07:42.983426 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 23:07:42.983672 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":103,"thread_start_us":102,"threads_started":1}
I20260810 23:07:42.983872 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:42.974537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57936 (local address 127.23.56.126:39999)
0810 23:07:42.974699 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:42.974709 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:42.974737 (+ 28us) server_negotiation.cc:407] Connection header received
0810 23:07:42.975279 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:42.975281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:42.975297 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:42.975377 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:42.975880 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:42.976682 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:42.977686 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:42.978195 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:42.978316 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:42.980635 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:42.980646 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:42.980646 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:42.980652 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:42.982209 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:42.982597 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:42.982600 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:42.982603 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:42.982678 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:42.982912 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:42.982913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:42.982913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:42.983172 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:42.983265 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:42.983474 (+ 209us) server_negotiation.cc:299] Negotiation successful
0810 23:07:42.983607 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":104,"threads_started":1}
I20260810 23:07:42.984593 16912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:42.984951 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:42.984827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:43417 (local address 127.0.0.1:53240)
0810 23:07:42.984877 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:42.984920 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260810 23:07:42.985083 16857 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
I20260810 23:07:42.985512 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:42.985409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:43417 (local address 127.0.0.1:53244)
0810 23:07:42.985466 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:42.985490 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 23:07:42.985668 16912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.125:43417: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.23.56.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab.
Entry for principal kudu/127.23.56.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab.
I20260810 23:07:43.006382 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.125:43417
--webserver_interface=127.23.56.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab
--principal=kudu/127.23.56.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:39999,127.23.56.125:43417,127.23.56.124:37525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:07:43.034963 16912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } attempt: 1
I20260810 23:07:43.035543 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.035394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:43417 (local address 127.0.0.1:53246)
0810 23:07:43.035471 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.035514 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 23:07:43.035903 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.035825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:43417 (local address 127.0.0.1:53256)
0810 23:07:43.035864 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.035888 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 23:07:43.036021 16912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.125:43417: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
W20260810 23:07:43.088907 16919 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:07:43.089107 16919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:43.089123 16919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:43.089143 16919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:43.090175 16919 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:43.090210 16919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:43.090224 16919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:43.090236 16919 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:43.090250 16919 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:43.091763 16919 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:39999,127.23.56.125:43417,127.23.56.124:37525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.125.keytab
--principal=kudu/127.23.56.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.125:43417
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.56.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:43.091948 16919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:43.092108 16919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:43.096858 16930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:43.096812 16927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:43.096805 16925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:43.097047 16919 server_base.cc:1034] running on GCE node
I20260810 23:07:43.097318 16919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:43.097568 16919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:43.098737 16919 hybrid_clock.cc:648] HybridClock initialized: now 1786403263098711 us; error 47 us; skew 500 ppm
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403263, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:43.101257 16919 init.cc:377] Logged in from keytab as kudu/127.23.56.125@KRBTEST.COM (short username kudu)
I20260810 23:07:43.101698 16919 webserver.cc:460] Webserver started at http://127.23.56.125:33079/ using document root <none> and password file <none>
I20260810 23:07:43.101914 16919 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:43.101984 16919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:43.102136 16919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:43.103222 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/data/instance:
uuid: "0901eecfa78f4678856e05d1e0e4e43e"
format_stamp: "Formatted at 2026-08-10 23:07:43 on dist-test-slave-1xrh"
server_key: "3c32874cf9a1e0067dc4a2c925350ca6"
server_key_iv: "ca1129ce32bdabd043c19e130fbb661f"
server_key_version: "encryptionkey@0"
I20260810 23:07:43.103583 16919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/wal/instance:
uuid: "0901eecfa78f4678856e05d1e0e4e43e"
format_stamp: "Formatted at 2026-08-10 23:07:43 on dist-test-slave-1xrh"
server_key: "3c32874cf9a1e0067dc4a2c925350ca6"
server_key_iv: "ca1129ce32bdabd043c19e130fbb661f"
server_key_version: "encryptionkey@0"
I20260810 23:07:43.104175 16912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } attempt: 2
I20260810 23:07:43.104554 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.104425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:43417 (local address 127.0.0.1:53266)
0810 23:07:43.104486 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.104521 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 23:07:43.104882 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.104787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:43417 (local address 127.0.0.1:53268)
0810 23:07:43.104826 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.104857 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260810 23:07:43.105008 16912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.125:43417: Network error: Client connection negotiation failed: client connection to 127.23.56.125:43417: connect: Connection refused (error 111)
I20260810 23:07:43.105093 16919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 23:07:43.105815 16938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:43.105979 16919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260810 23:07:43.106076 16919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/wal
uuid: "0901eecfa78f4678856e05d1e0e4e43e"
format_stamp: "Formatted at 2026-08-10 23:07:43 on dist-test-slave-1xrh"
server_key: "3c32874cf9a1e0067dc4a2c925350ca6"
server_key_iv: "ca1129ce32bdabd043c19e130fbb661f"
server_key_version: "encryptionkey@0"
I20260810 23:07:43.106155 16919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:43.121851 16919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:43.123741 16919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:43.123903 16919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:43.127838 16919 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.125:43417
I20260810 23:07:43.127885 16992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.125:43417 every 8 connection(s)
I20260810 23:07:43.128247 16919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/data/info.pb
I20260810 23:07:43.129200 16993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:43.130625 16993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:43.131587 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 16919
I20260810 23:07:43.131692 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-1/wal/instance
I20260810 23:07:43.131891 23777 external_mini_cluster.cc:1428] Setting key 1618ad66d38bca2c57ee88e30f1f268c
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403263, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.56.124@KRBTEST.COM; defaulting to no policy
I20260810 23:07:43.140324 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.131303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57950 (local address 127.23.56.126:39999)
0810 23:07:43.131401 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.131404 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.131674 (+ 270us) server_negotiation.cc:407] Connection header received
0810 23:07:43.132348 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.132350 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.132363 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.132409 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.133212 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.133803 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.134739 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.134976 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.135035 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.137399 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.137401 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.137401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.137409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.138832 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.139348 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.139349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.139350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.139417 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.139672 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.139674 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.139674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.139791 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.139872 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.140142 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.140231 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 23:07:43.140520 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.131280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:39999 (local address 127.0.0.1:57950)
0810 23:07:43.131510 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.131562 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:43.132232 (+ 670us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:43.132481 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:43.132486 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:43.132864 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:43.133127 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.133133 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.133898 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.133899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.134663 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.134665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.135092 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.135094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.135252 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.135696 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:43.135706 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:43.137285 (+ 1579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:43.138954 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.138961 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.138963 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.139271 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.139487 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.139488 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.139489 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.139603 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.139863 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:43.139870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:43.140010 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 23:07:43.140119 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 23:07:43.140274 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":101,"threads_started":1}
I20260810 23:07:43.141041 16993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.56.124@KRBTEST.COM" created.
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403263, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
I20260810 23:07:43.148629 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.141316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:43417 (local address 127.0.0.1:53278)
0810 23:07:43.141399 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.141411 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:43.141464 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:43.141682 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:43.141683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:43.141791 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:43.141835 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.141836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.142710 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.142710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.143370 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.143373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.143856 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.143857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.143937 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.144372 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:43.144376 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:43.145638 (+ 1262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:43.147317 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.147318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.147319 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.147550 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.147830 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.147830 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.147831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.147881 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.148262 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:43.148263 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:43.148303 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 23:07:43.148375 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 23:07:43.148472 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 23:07:43.148661 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.141380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53278 (local address 127.23.56.125:43417)
0810 23:07:43.141550 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.141554 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.141571 (+ 17us) server_negotiation.cc:407] Connection header received
0810 23:07:43.141589 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.141591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.141602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.141666 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.141897 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.142629 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.143460 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.143757 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.143848 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.145729 (+ 1881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.145738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.145738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.145751 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.147256 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.147681 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.147684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.147685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.147755 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.147941 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.147960 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.147960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.148146 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.148236 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.148460 (+ 224us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.148545 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":93,"threads_started":1}
I20260810 23:07:43.150909 16993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:43.151335 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.151184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43236)
0810 23:07:43.151221 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.151304 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260810 23:07:43.151533 16941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
I20260810 23:07:43.151969 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.151879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43238)
0810 23:07:43.151910 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.151943 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260810 23:07:43.155172 16993 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:37525: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.56.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab.
Entry for principal kudu/127.23.56.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab.
I20260810 23:07:43.162235 23777 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
/tmp/dist-test-taskQ_MaNL/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.124:37525
--webserver_interface=127.23.56.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab
--principal=kudu/127.23.56.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.56.84:45013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.56.126:39999,127.23.56.125:43417,127.23.56.124:37525
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 23:07:43.196159 16912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } attempt: 3
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403262, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
I20260810 23:07:43.203852 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.196612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.23.56.125:43417)
0810 23:07:43.196691 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.196693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.196704 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:07:43.196733 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.196735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.196747 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.196817 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.197162 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.197686 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.198383 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.198617 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.198705 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.200917 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.200919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.200919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.200929 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.202410 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.202850 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.202852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.202853 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.202920 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.203234 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.203235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.203235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.203393 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.203472 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.203646 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.203728 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 23:07:43.203859 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.196512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:43417 (local address 127.0.0.1:53292)
0810 23:07:43.196584 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.196604 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:43.196673 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:43.196824 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:43.196825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:43.197007 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:43.197073 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.197075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.197780 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.197781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.198296 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.198297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.198705 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.198706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.198796 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.199324 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:43.199329 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:43.200758 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:43.202518 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.202519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.202520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.202719 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.203000 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.203001 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.203002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.203097 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.203476 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:43.203478 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:43.203525 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 23:07:43.203618 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 23:07:43.203726 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 23:07:43.204290 16912 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:43.204430 16993 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } attempt: 1
I20260810 23:07:43.204648 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.204510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43246)
0810 23:07:43.204569 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.204610 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 23:07:43.204777 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.204680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43252)
0810 23:07:43.204729 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.204752 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260810 23:07:43.205001 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.204919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43256)
0810 23:07:43.204954 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.204979 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260810 23:07:43.205120 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.205011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43272)
0810 23:07:43.205067 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.205097 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260810 23:07:43.205256 16993 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:37525: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
W20260810 23:07:43.205258 16912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:37525: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
I20260810 23:07:43.236413 16912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } attempt: 1
I20260810 23:07:43.236867 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.236736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43274)
0810 23:07:43.236801 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.236838 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260810 23:07:43.237277 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.237177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43290)
0810 23:07:43.237236 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.237260 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260810 23:07:43.237366 16912 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:37525: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
W20260810 23:07:43.246851 17002 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 23:07:43.247059 17002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 23:07:43.247113 17002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 23:07:43.247175 17002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 23:07:43.248339 17002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 23:07:43.248400 17002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 23:07:43.248426 17002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 23:07:43.248453 17002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 23:07:43.248483 17002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 23:07:43.249948 17002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.56.84:45013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.56.126:39999,127.23.56.125:43417,127.23.56.124:37525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/krb5kdc/kudu_127.23.56.124.keytab
--principal=kudu/127.23.56.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.56.124:37525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.56.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 10 Aug 2026 22:08:11 UTC on ccycloud.quasar-zbnufz.root.comops.site
build id 4579241
I20260810 23:07:43.250221 17002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 23:07:43.250438 17002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 23:07:43.255098 17013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:43.255291 17002 server_base.cc:1034] running on GCE node
W20260810 23:07:43.255102 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 23:07:43.255093 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 23:07:43.255632 17002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 23:07:43.255865 17002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 23:07:43.257005 17002 hybrid_clock.cc:648] HybridClock initialized: now 1786403263256974 us; error 44 us; skew 500 ppm
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403263, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 23:07:43.259536 17002 init.cc:377] Logged in from keytab as kudu/127.23.56.124@KRBTEST.COM (short username kudu)
I20260810 23:07:43.260007 17002 webserver.cc:460] Webserver started at http://127.23.56.124:38257/ using document root <none> and password file <none>
I20260810 23:07:43.260236 17002 fs_manager.cc:359] Metadata directory not provided
I20260810 23:07:43.260303 17002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 23:07:43.260463 17002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 23:07:43.261507 17002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/data/instance:
uuid: "ca41701a60fc4e2fb79011ba8e3c3928"
format_stamp: "Formatted at 2026-08-10 23:07:43 on dist-test-slave-1xrh"
server_key: "3981e6bebb49f31a40bcccc713e2f6ee"
server_key_iv: "eb375911f9d4d52be9bace59bd163ce7"
server_key_version: "encryptionkey@0"
I20260810 23:07:43.261857 17002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/wal/instance:
uuid: "ca41701a60fc4e2fb79011ba8e3c3928"
format_stamp: "Formatted at 2026-08-10 23:07:43 on dist-test-slave-1xrh"
server_key: "3981e6bebb49f31a40bcccc713e2f6ee"
server_key_iv: "eb375911f9d4d52be9bace59bd163ce7"
server_key_version: "encryptionkey@0"
I20260810 23:07:43.263118 17002 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260810 23:07:43.263890 17021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 23:07:43.264073 17002 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260810 23:07:43.264169 17002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/data,/tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/wal
uuid: "ca41701a60fc4e2fb79011ba8e3c3928"
format_stamp: "Formatted at 2026-08-10 23:07:43 on dist-test-slave-1xrh"
server_key: "3981e6bebb49f31a40bcccc713e2f6ee"
server_key_iv: "eb375911f9d4d52be9bace59bd163ce7"
server_key_version: "encryptionkey@0"
I20260810 23:07:43.264256 17002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 23:07:43.273401 16993 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } attempt: 2
I20260810 23:07:43.273808 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.273687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43304)
0810 23:07:43.273746 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.273780 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260810 23:07:43.274161 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.274068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43316)
0810 23:07:43.274111 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.274136 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 23:07:43.274250 16993 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.56.124:37525: Network error: Client connection negotiation failed: client connection to 127.23.56.124:37525: connect: Connection refused (error 111)
I20260810 23:07:43.279305 17002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 23:07:43.281503 17002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 23:07:43.281662 17002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 23:07:43.285778 17002 rpc_server.cc:307] RPC server started. Bound to: 127.23.56.124:37525
I20260810 23:07:43.285812 17078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.56.124:37525 every 8 connection(s)
I20260810 23:07:43.286135 17002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/data/info.pb
I20260810 23:07:43.287168 17079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 23:07:43.287165 23777 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu as pid 17002
I20260810 23:07:43.287290 23777 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskQ_MaNL/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786402992661514-23777-0/minicluster-data/master-2/wal/instance
I20260810 23:07:43.287482 23777 external_mini_cluster.cc:1428] Setting key 13abcc949163d9306a96e6ed39c8dcc4
I20260810 23:07:43.289506 17079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403263, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:43.298344 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.290213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57954 (local address 127.23.56.126:39999)
0810 23:07:43.290283 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.290285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.290457 (+ 172us) server_negotiation.cc:407] Connection header received
0810 23:07:43.290515 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.290516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.290525 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.290571 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.291094 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.291820 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.292890 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.293088 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.293173 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.295516 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.295518 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.295518 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.295526 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.296983 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.297405 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.297406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.297407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.297465 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.297705 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.297707 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.297707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.297842 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.297917 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.298168 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.298254 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 23:07:43.298506 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.287982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43320 (local address 127.23.56.124:37525)
0810 23:07:43.288190 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.288199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.288232 (+ 33us) server_negotiation.cc:407] Connection header received
0810 23:07:43.288785 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.288795 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.288860 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.289020 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.289606 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.290923 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.291993 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.292556 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.292666 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.295096 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.295104 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.295105 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.295112 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.296817 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.297269 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.297272 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.297272 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.297348 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.297597 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.297598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.297598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.297870 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.297992 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.298167 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.298283 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":94,"threads_started":1}
I20260810 23:07:43.298748 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.290166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.126:39999 (local address 127.0.0.1:57954)
0810 23:07:43.290338 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.290376 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:43.290455 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:43.290618 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:43.290621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:43.290939 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:43.291018 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.291021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.291920 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.291921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.292804 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.292807 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.293208 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.293210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.293284 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.293729 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:43.293737 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:43.295428 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:43.297060 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.297066 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.297068 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.297332 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.297534 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.297535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.297535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.297631 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.297954 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:43.297962 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:43.298026 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 23:07:43.298122 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 23:07:43.298678 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":88,"thread_start_us":100,"threads_started":1}
I20260810 23:07:43.299314 17079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403263, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.126@KRBTEST.COM
I20260810 23:07:43.305896 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.299555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.125:43417 (local address 127.0.0.1:53306)
0810 23:07:43.299616 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.299625 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:43.299654 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:43.299787 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:43.299788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:43.299895 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:43.299921 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.299922 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.300575 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.300577 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.301075 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.301077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.301511 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.301512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.301564 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.301970 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:43.301975 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:43.303089 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:43.304883 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.304884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.304884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.305068 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.305265 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.305266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.305267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.305322 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.305572 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:43.305573 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:43.305612 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 23:07:43.305698 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 23:07:43.305804 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 23:07:43.305896 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.299628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.23.56.125:43417)
0810 23:07:43.299670 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.299672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.299680 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:43.299707 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.299708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.299719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.299787 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.299991 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.300499 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.301156 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.301413 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.301496 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.303206 (+ 1710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.303208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.303208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.303215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.304775 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.305141 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.305142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.305142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.305201 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.305393 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.305394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.305395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.305508 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.305578 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.305723 (+ 145us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.305809 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:07:43.306280 17079 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } has no permanent_uuid. Determining permanent_uuid...
I20260810 23:07:43.306668 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.299336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57966 (local address 127.23.56.126:39999)
0810 23:07:43.299373 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.299375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.299386 (+ 11us) server_negotiation.cc:407] Connection header received
0810 23:07:43.299504 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.299505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.299517 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.299572 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.300131 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.300574 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.301231 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.301398 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.301449 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.303809 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.303812 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.303812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.303823 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.305245 (+ 1422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.305572 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.305574 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.305574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.305628 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.305848 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.305849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.305850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.306102 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.306270 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.306474 (+ 204us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.306576 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403263, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.124@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
I20260810 23:07:43.312520 16912 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } attempt: 2
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403262, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.56.125@KRBTEST.COM
I20260810 23:07:43.313676 17085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.306505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43330)
0810 23:07:43.306776 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.306787 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:43.306845 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:43.306982 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:43.306982 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:43.307083 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:43.307107 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.307108 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.307920 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.307921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.308591 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.308594 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.309054 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.309055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.309105 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.309466 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:43.309471 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:43.310659 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:43.312681 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.312682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.312682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.312883 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.313063 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.313064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.313065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.313117 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.313343 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:43.313344 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:43.313385 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 23:07:43.313466 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 23:07:43.313577 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255}
I20260810 23:07:43.313822 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.306575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43330 (local address 127.23.56.124:37525)
0810 23:07:43.306606 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.306608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.306852 (+ 244us) server_negotiation.cc:407] Connection header received
0810 23:07:43.306883 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.306884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.306895 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.306970 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.307213 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.307846 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.308660 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.308977 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.309046 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.310746 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.310747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.310747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.310758 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.312599 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.312953 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.312954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.312954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.313003 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.313177 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.313178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.313178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.313279 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.313340 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.313515 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.313642 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":17}
I20260810 23:07:43.315485 17079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928: Bootstrap starting.
I20260810 23:07:43.316342 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.307613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.23.56.125:43417)
0810 23:07:43.307658 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.307660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.307680 (+ 20us) server_negotiation.cc:407] Connection header received
0810 23:07:43.307699 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.307700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.307708 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.307760 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.308331 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.308977 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.309956 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.310417 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.310554 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.313298 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.313300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.313300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.313307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.315021 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.315446 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.315447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.315448 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.315504 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.315726 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.315726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.315727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.315851 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.316019 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.316174 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.316261 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260810 23:07:43.316464 17079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:43.316758 17079 log.cc:824] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:43.318050 17079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928: No bootstrap required, opened a new log
I20260810 23:07:43.320230 23777 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 23:07:43.320219 17079 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } }
I20260810 23:07:43.320360 17079 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca41701a60fc4e2fb79011ba8e3c3928, State: Initialized, Role: FOLLOWER
I20260810 23:07:43.320461 17079 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } }
I20260810 23:07:43.320786 17079 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:43.320772 17088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.320883 17088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [sys.catalog]: This master's current role is: FOLLOWER
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403262, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.126@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
I20260810 23:07:43.321266 16884 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5122cb5c44274f288700b415081ecf7f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57966
W20260810 23:07:43.322715 17102 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:07:43.322966 17102 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:07:43.325157 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.313073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43344)
0810 23:07:43.314669 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.314679 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:43.314730 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:43.317177 (+ 2447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:43.317178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:43.317354 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:43.317394 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.317395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.318270 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.318271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.319002 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.319005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.319519 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.319520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.319574 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.319993 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:43.319998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:43.322013 (+ 2015us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:43.324032 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.324034 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.324034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.324248 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.324481 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.324481 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.324482 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.324546 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.324842 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:43.324843 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:43.324903 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 23:07:43.324976 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 23:07:43.325067 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1577}
I20260810 23:07:43.325173 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.312833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43344 (local address 127.23.56.124:37525)
0810 23:07:43.314284 (+ 1451us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.314286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.317077 (+ 2791us) server_negotiation.cc:407] Connection header received
0810 23:07:43.317096 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.317096 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.317104 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.317169 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.317490 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.318177 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.319103 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.319424 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.319713 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.322096 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.322097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.322098 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.322104 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.323943 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.324331 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.324332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.324333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.324403 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.324612 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.324613 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.324614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.324767 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.324826 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.325015 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.325095 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1155,"thread_start_us":309,"threads_started":1}
I20260810 23:07:43.323051 17002 master_runner.cc:186] Error getting master registration for 127.23.56.126:39999: OK, instance_id { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" instance_seqno: 1786403262969677 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:07:43.325861 17002 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:07:43.326653 16912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f: Bootstrap starting.
I20260810 23:07:43.327381 16912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:43.327690 16912 log.cc:824] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:43.328548 16912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f: No bootstrap required, opened a new log
I20260810 23:07:43.329874 16912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } }
I20260810 23:07:43.330015 16912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5122cb5c44274f288700b415081ecf7f, State: Initialized, Role: FOLLOWER
I20260810 23:07:43.330098 16912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } }
I20260810 23:07:43.330404 16912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:43.330456 16884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 23:07:43.330525 16884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:43.331318 16884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } }
I20260810 23:07:43.331576 16884 leader_election.cc:288] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [CANDIDATE]: Term 1 election: Requested vote from peers 0901eecfa78f4678856e05d1e0e4e43e (127.23.56.125:43417), ca41701a60fc4e2fb79011ba8e3c3928 (127.23.56.124:37525)
I20260810 23:07:43.332017 17051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5122cb5c44274f288700b415081ecf7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ca41701a60fc4e2fb79011ba8e3c3928"
I20260810 23:07:43.332125 17051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:43.333307 17051 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5122cb5c44274f288700b415081ecf7f in term 1.
I20260810 23:07:43.333592 16857 leader_election.cc:302] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5122cb5c44274f288700b415081ecf7f, ca41701a60fc4e2fb79011ba8e3c3928; no voters:
I20260810 23:07:43.333839 17115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 23:07:43.330374 17105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.334195 17105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:43.334487 17115 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [term 1 LEADER]: Becoming Leader. State: Replica: 5122cb5c44274f288700b415081ecf7f, State: Running, Role: LEADER
I20260810 23:07:43.334715 17115 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } }
I20260810 23:07:43.336011 17105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5122cb5c44274f288700b415081ecf7f. Latest consensus state: current_term: 1 leader_uuid: "5122cb5c44274f288700b415081ecf7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.336726 17105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [sys.catalog]: This master's current role is: LEADER
W20260810 23:07:43.336671 17128 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 23:07:43.336915 17128 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 23:07:43.337456 16834 master_runner.cc:186] Error getting master registration for 127.23.56.125:43417: OK, instance_id { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" instance_seqno: 1786403263126693 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 23:07:43.339823 16834 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 23:07:43.339911 17130 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 23:07:43.340112 17130 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 23:07:43.341948 17051 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [term 1 FOLLOWER]: Refusing update from remote peer 5122cb5c44274f288700b415081ecf7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:43.343370 17105 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:43.345281 17088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5122cb5c44274f288700b415081ecf7f. Latest consensus state: current_term: 1 leader_uuid: "5122cb5c44274f288700b415081ecf7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.345389 17088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:43.345948 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.334781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43350 (local address 127.23.56.124:37525)
0810 23:07:43.334822 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.334824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.335092 (+ 268us) server_negotiation.cc:407] Connection header received
0810 23:07:43.335233 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.335234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.335248 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.335286 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.337420 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.338132 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.339181 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.339460 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.339505 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.341842 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.341844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.341845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.341852 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.343657 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.344803 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.344804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.344804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.344857 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.345052 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.345053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.345053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.345173 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.345250 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.345780 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.345853 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:07:43.347170 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.334566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53316 (local address 127.23.56.125:43417)
0810 23:07:43.334607 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.334609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.335240 (+ 631us) server_negotiation.cc:407] Connection header received
0810 23:07:43.335420 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.335421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.335451 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.335482 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.336850 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.337687 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.338787 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.339582 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.339628 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.342131 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.342133 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.342133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.342141 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.344416 (+ 2275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.346234 (+ 1818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.346235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.346236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.346296 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.346516 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.346517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.346517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.346640 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.346704 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.346954 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.347036 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:07:43.349484 16915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.337194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.23.56.126:39999)
0810 23:07:43.337231 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.337233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.341500 (+ 4267us) server_negotiation.cc:407] Connection header received
0810 23:07:43.341605 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.341606 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.341615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.341646 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.342307 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.343024 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.344075 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.344332 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.344376 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.345931 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.345947 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.345947 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.345954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.348210 (+ 2256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.348656 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.348658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.348658 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.348713 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.348919 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.348920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.348920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.349039 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.349113 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.349319 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.349394 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:07:43.354274 17115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5122cb5c44274f288700b415081ecf7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.354404 17115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:43.354480 17132 mvcc.cc:205] Tried to move back new op lower bound from 7317107766645690368 to 7317107766621319168. Current Snapshot: MvccSnapshot[applied={T|T < 7317107766645690368}]
I20260810 23:07:43.354521 17088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5122cb5c44274f288700b415081ecf7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.354606 17088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:43.356386 17130 catalog_manager.cc:1347] Generated new cluster ID: 1325ec73b8124321b4c9e2ba81841206
I20260810 23:07:43.356442 17130 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 23:07:43.365312 16993 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } attempt: 3
I20260810 23:07:43.367079 17130 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 23:07:43.367777 17130 catalog_manager.cc:1504] Loading token signing keys...
Aug 10 23:07:43 dist-test-slave-1xrh krb5kdc[16821](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786403263, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.56.125@KRBTEST.COM for kudu/127.23.56.124@KRBTEST.COM
I20260810 23:07:43.372893 17130 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f: Generated new TSK 0
I20260810 23:07:43.373066 17130 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 23:07:43.374372 16995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.365547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.56.124:37525 (local address 127.0.0.1:43364)
0810 23:07:43.365628 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0810 23:07:43.365640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 23:07:43.365679 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 23:07:43.365938 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 23:07:43.365939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 23:07:43.366110 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 23:07:43.366153 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.366155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.367069 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.367070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.367982 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 23:07:43.367985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.368594 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.368595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 23:07:43.368654 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.369170 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 23:07:43.369175 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 23:07:43.370648 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 23:07:43.373226 (+ 2578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.373227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.373228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.373456 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.373694 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 23:07:43.373695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 23:07:43.373696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 23:07:43.373753 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 23:07:43.374073 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 23:07:43.374074 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 23:07:43.374132 (+ 58us) client_negotiation.cc:769] Sending connection context
0810 23:07:43.374179 (+ 47us) client_negotiation.cc:240] Negotiation successful
0810 23:07:43.374272 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 23:07:43.375013 17084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.365759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.23.56.124:37525)
0810 23:07:43.365800 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.365802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.365809 (+ 7us) server_negotiation.cc:407] Connection header received
0810 23:07:43.365827 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.365828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.365838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.365889 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.366234 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.366985 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.368167 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.368479 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.368541 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.370734 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 23:07:43.370736 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 23:07:43.370736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 23:07:43.370743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0810 23:07:43.373058 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.373574 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.373575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.373575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.373629 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 23:07:43.373837 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 23:07:43.373837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 23:07:43.373838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 23:07:43.373976 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 23:07:43.374793 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.374835 (+ 42us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.374926 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260810 23:07:43.376708 16993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e: Bootstrap starting.
I20260810 23:07:43.377425 16993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e: Neither blocks nor log segments found. Creating new log.
I20260810 23:07:43.377720 16993 log.cc:824] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e: Log is configured to *not* fsync() on all Append() calls
I20260810 23:07:43.378537 16993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e: No bootstrap required, opened a new log
I20260810 23:07:43.380573 16993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } }
I20260810 23:07:43.380705 16993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0901eecfa78f4678856e05d1e0e4e43e, State: Initialized, Role: FOLLOWER
I20260810 23:07:43.380780 16993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } }
I20260810 23:07:43.381098 16993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 23:07:43.383252 16966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5122cb5c44274f288700b415081ecf7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0901eecfa78f4678856e05d1e0e4e43e"
I20260810 23:07:43.383363 16966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [term 0 FOLLOWER]: Advancing to term 1
I20260810 23:07:43.384179 16966 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5122cb5c44274f288700b415081ecf7f in term 1.
I20260810 23:07:43.384579 16965 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [term 1 FOLLOWER]: Refusing update from remote peer 5122cb5c44274f288700b415081ecf7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 23:07:43.384871 17124 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 23:07:43.386300 17152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5122cb5c44274f288700b415081ecf7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.386399 17152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:43.386445 17152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5122cb5c44274f288700b415081ecf7f. Latest consensus state: current_term: 1 leader_uuid: "5122cb5c44274f288700b415081ecf7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.386494 17152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:43.386528 17152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5122cb5c44274f288700b415081ecf7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.386572 17152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 23:07:43.387080 17153 mvcc.cc:205] Tried to move back new op lower bound from 7317107766770659328 to 7317107766621319168. Current Snapshot: MvccSnapshot[applied={T|T < 7317107766645690368}]
I20260810 23:07:43.390548 17151 catalog_manager.cc:1259] Loaded cluster ID: 1325ec73b8124321b4c9e2ba81841206
I20260810 23:07:43.390604 17151 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e: loading cluster ID for follower catalog manager: success
I20260810 23:07:43.391793 17151 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e: acquiring CA information for follower catalog manager: success
I20260810 23:07:43.392093 17151 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0901eecfa78f4678856e05d1e0e4e43e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 23:07:43.408754 16998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.407052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.23.56.125:43417)
0810 23:07:43.407094 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.407096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.407989 (+ 893us) server_negotiation.cc:407] Connection header received
0810 23:07:43.408099 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.408100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.408111 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.408162 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.408693 (+ 531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53324: BlockingRecv error: recv got EOF from 127.0.0.1:53324 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 23:07:43.409731 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.407474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43376 (local address 127.23.56.124:37525)
0810 23:07:43.407512 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.407514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.409019 (+ 1505us) server_negotiation.cc:407] Connection header received
0810 23:07:43.409116 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.409117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.409127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.409157 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.409674 (+ 517us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43376: BlockingRecv error: recv got EOF from 127.0.0.1:43376 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260810 23:07:43.410570 16915 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.407852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57980 (local address 127.23.56.126:39999)
0810 23:07:43.407896 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.407899 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.410012 (+ 2113us) server_negotiation.cc:407] Connection header received
0810 23:07:43.410116 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.410118 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.410128 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.410159 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.410495 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57980: BlockingRecv error: recv got EOF from 127.0.0.1:57980 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
I20260810 23:07:43.415804 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.412253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53338 (local address 127.23.56.125:43417)
0810 23:07:43.412300 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.412302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.412523 (+ 221us) server_negotiation.cc:407] Connection header received
0810 23:07:43.412634 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.412635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.412670 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.412715 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.412933 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.413686 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.414816 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.415109 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.415186 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.415423 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.415537 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.415585 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.415706 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.415740 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260810 23:07:43.418872 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.418130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.23.56.124:37525)
0810 23:07:43.418166 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.418168 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.418370 (+ 202us) server_negotiation.cc:407] Connection header received
0810 23:07:43.418468 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.418469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.418486 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.418515 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.418828 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43382: BlockingRecv error: recv got EOF from 127.0.0.1:43382 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:07:43.424510 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.421267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53340 (local address 127.23.56.125:43417)
0810 23:07:43.421315 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.421316 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.421567 (+ 251us) server_negotiation.cc:407] Connection header received
0810 23:07:43.421716 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.421717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.421727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.421757 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.421937 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.422605 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.423685 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.423974 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.424023 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.424234 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.424268 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.424312 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.424426 (+ 114us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.424447 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260810 23:07:43.427508 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.426721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.23.56.124:37525)
0810 23:07:43.426756 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.426757 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.426967 (+ 210us) server_negotiation.cc:407] Connection header received
0810 23:07:43.427064 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.427065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.427076 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.427105 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.427464 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43388: BlockingRecv error: recv got EOF from 127.0.0.1:43388 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:07:43.433758 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.430549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53352 (local address 127.23.56.125:43417)
0810 23:07:43.430616 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.430618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.430867 (+ 249us) server_negotiation.cc:407] Connection header received
0810 23:07:43.430963 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.430964 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.430973 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.431001 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.431204 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.431899 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.432971 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.433236 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.433296 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.433497 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.433527 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.433555 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.433681 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.433697 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 23:07:43.436669 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.435894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43396 (local address 127.23.56.124:37525)
0810 23:07:43.435929 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.435931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.436137 (+ 206us) server_negotiation.cc:407] Connection header received
0810 23:07:43.436243 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.436244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.436254 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.436295 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.436626 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43396: BlockingRecv error: recv got EOF from 127.0.0.1:43396 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
I20260810 23:07:43.444990 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.441676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53362 (local address 127.23.56.125:43417)
0810 23:07:43.441710 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.441712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.441940 (+ 228us) server_negotiation.cc:407] Connection header received
0810 23:07:43.442034 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.442035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.442044 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.442072 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.442249 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.442967 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.444144 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.444428 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.444475 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.444739 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.444770 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.444800 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.444910 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.444929 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 23:07:43.447837 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.446996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.23.56.124:37525)
0810 23:07:43.447030 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.447032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.447336 (+ 304us) server_negotiation.cc:407] Connection header received
0810 23:07:43.447435 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.447436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.447447 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.447475 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.447790 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43406: BlockingRecv error: recv got EOF from 127.0.0.1:43406 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260810 23:07:43.460140 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.456849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53376 (local address 127.23.56.125:43417)
0810 23:07:43.456916 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.456918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.457168 (+ 250us) server_negotiation.cc:407] Connection header received
0810 23:07:43.457260 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.457261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.457271 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.457312 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.457491 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.458197 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.459355 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.459641 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.459682 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.459881 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.459914 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.459942 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.460044 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.460065 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260810 23:07:43.463241 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.462421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43416 (local address 127.23.56.124:37525)
0810 23:07:43.462459 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.462460 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.462657 (+ 197us) server_negotiation.cc:407] Connection header received
0810 23:07:43.462768 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.462769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.462782 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.462814 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.463195 (+ 381us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43416: BlockingRecv error: recv got EOF from 127.0.0.1:43416 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:07:43.484036 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.480507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.23.56.125:43417)
0810 23:07:43.480548 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.480550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.480837 (+ 287us) server_negotiation.cc:407] Connection header received
0810 23:07:43.480936 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.480938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.480948 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.480976 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.481259 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.482000 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.483232 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.483525 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.483563 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.483755 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.483810 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.483837 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.483933 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.483952 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260810 23:07:43.487037 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.486242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43428 (local address 127.23.56.124:37525)
0810 23:07:43.486280 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.486282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.486476 (+ 194us) server_negotiation.cc:407] Connection header received
0810 23:07:43.486576 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.486578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.486590 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.486618 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.486983 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43428: BlockingRecv error: recv got EOF from 127.0.0.1:43428 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260810 23:07:43.523265 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.520302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.23.56.125:43417)
0810 23:07:43.520351 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.520353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.520510 (+ 157us) server_negotiation.cc:407] Connection header received
0810 23:07:43.520585 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.520587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.520598 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.520679 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.520975 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.521565 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.522512 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.522725 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.522802 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.522967 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.522999 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.523062 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.523149 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.523168 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260810 23:07:43.525822 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.525025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.23.56.124:37525)
0810 23:07:43.525070 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.525072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.525312 (+ 240us) server_negotiation.cc:407] Connection header received
0810 23:07:43.525390 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.525391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.525404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.525454 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.525767 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43430: BlockingRecv error: recv got EOF from 127.0.0.1:43430 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:07:43.593501 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.591240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53400 (local address 127.23.56.125:43417)
0810 23:07:43.591314 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.591316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.591457 (+ 141us) server_negotiation.cc:407] Connection header received
0810 23:07:43.591501 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.591502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.591511 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.591578 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.591702 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.592173 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.592921 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.593107 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.593167 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.593296 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.593325 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.593371 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.593422 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.593439 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 23:07:43.595920 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.595255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43442 (local address 127.23.56.124:37525)
0810 23:07:43.595327 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.595329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.595403 (+ 74us) server_negotiation.cc:407] Connection header received
0810 23:07:43.595484 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.595485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.595496 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.595543 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.595859 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43442: BlockingRecv error: recv got EOF from 127.0.0.1:43442 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 23:07:43.727334 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.725029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53402 (local address 127.23.56.125:43417)
0810 23:07:43.725128 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.725130 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.725148 (+ 18us) server_negotiation.cc:407] Connection header received
0810 23:07:43.725232 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.725233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.725244 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.725303 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.725420 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.725933 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.726632 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.726849 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.726905 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.727037 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.727065 (+ 28us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.727104 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.727212 (+ 108us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.727229 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260810 23:07:43.729770 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.729139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43456 (local address 127.23.56.124:37525)
0810 23:07:43.729210 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.729213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.729257 (+ 44us) server_negotiation.cc:407] Connection header received
0810 23:07:43.729332 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.729334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.729345 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.729395 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.729711 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43456: BlockingRecv error: recv got EOF from 127.0.0.1:43456 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 23:07:43.909348 17124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5122cb5c44274f288700b415081ecf7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5122cb5c44274f288700b415081ecf7f" member_type: VOTER last_known_addr { host: "127.23.56.126" port: 39999 } } peers { permanent_uuid: "0901eecfa78f4678856e05d1e0e4e43e" member_type: VOTER last_known_addr { host: "127.23.56.125" port: 43417 } } peers { permanent_uuid: "ca41701a60fc4e2fb79011ba8e3c3928" member_type: VOTER last_known_addr { host: "127.23.56.124" port: 37525 } } }
I20260810 23:07:43.909540 17124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5122cb5c44274f288700b415081ecf7f [sys.catalog]: This master's current role is: LEADER
I20260810 23:07:43.989655 16998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:43.987202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53408 (local address 127.23.56.125:43417)
0810 23:07:43.987305 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.987308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.987380 (+ 72us) server_negotiation.cc:407] Connection header received
0810 23:07:43.987469 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.987470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.987485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.987560 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:43.987760 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.988259 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.989022 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:43.989209 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:43.989268 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:43.989405 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:43.989434 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:43.989473 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:43.989530 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 23:07:43.989546 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 23:07:43.991968 17084 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 23:07:43.991435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43458 (local address 127.23.56.124:37525)
0810 23:07:43.991494 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:43.991496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:43.991506 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:43.991534 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:43.991535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:43.991544 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:43.991591 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 23:07:43.991911 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43458: BlockingRecv error: recv got EOF from 127.0.0.1:43458 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 23:07:44.323478 17102 catalog_manager.cc:1259] Loaded cluster ID: 1325ec73b8124321b4c9e2ba81841206
I20260810 23:07:44.323559 17102 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928: loading cluster ID for follower catalog manager: success
I20260810 23:07:44.324385 17102 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928: acquiring CA information for follower catalog manager: success
I20260810 23:07:44.324708 17102 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ca41701a60fc4e2fb79011ba8e3c3928: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 23:07:44.507534 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:44.505115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53422 (local address 127.23.56.125:43417)
0810 23:07:44.505249 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:44.505251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:44.505263 (+ 12us) server_negotiation.cc:407] Connection header received
0810 23:07:44.505341 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:44.505342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:44.505354 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:44.505408 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:44.505538 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.506087 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.506888 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.507091 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.507184 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:44.507322 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:44.507349 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:44.507387 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:44.507449 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 23:07:44.507466 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260810 23:07:44.511510 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:44.509284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.23.56.124:37525)
0810 23:07:44.509402 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:44.509404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:44.509414 (+ 10us) server_negotiation.cc:407] Connection header received
0810 23:07:44.509463 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:44.509463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:44.509515 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:44.509575 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:44.509676 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.510221 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.510810 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.510995 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.511038 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:44.511202 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:44.511303 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:44.511354 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:44.511385 (+ 31us) server_negotiation.cc:299] Negotiation successful
0810 23:07:44.511402 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":71,"threads_started":1}
I20260810 23:07:44.515697 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:44.513073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.23.56.125:43417)
0810 23:07:44.513122 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:44.513124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:44.513308 (+ 184us) server_negotiation.cc:407] Connection header received
0810 23:07:44.513378 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:44.513379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:44.513389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:44.513437 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:44.513606 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.514062 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.514896 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.515196 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.515257 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:44.515423 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:44.515452 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:44.515479 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:44.515592 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 23:07:44.515620 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 23:07:44.515956 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:44.513238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.23.56.124:37525)
0810 23:07:44.513279 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:44.513281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:44.513424 (+ 143us) server_negotiation.cc:407] Connection header received
0810 23:07:44.513473 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:44.513474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:44.513484 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:44.513531 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:44.513672 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.514083 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.515240 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.515439 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.515524 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:44.515674 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:44.515703 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:44.515732 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:44.515866 (+ 134us) server_negotiation.cc:299] Negotiation successful
0810 23:07:44.515890 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260810 23:07:44.516956 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 23:07:44.513140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57984 (local address 127.23.56.126:39999)
0810 23:07:44.513301 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 23:07:44.513304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 23:07:44.513312 (+ 8us) server_negotiation.cc:407] Connection header received
0810 23:07:44.513357 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 23:07:44.513358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 23:07:44.513437 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 23:07:44.513527 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 23:07:44.514200 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.514972 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.516144 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 23:07:44.516483 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 23:07:44.516548 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 23:07:44.516673 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 23:07:44.516746 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 23:07:44.516787 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 23:07:44.516835 (+ 48us) server_negotiation.cc:299] Negotiation successful
0810 23:07:44.516851 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":51,"threads_started":1}
I20260810 23:07:44.524937 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 16834
I20260810 23:07:44.532475 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 16919
I20260810 23:07:44.539464 23777 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskQ_MaNL/build/release/bin/kudu with pid 17002
2026-08-10T23:07:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2905 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5757 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271887 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST